Disk not being calculated, just stuck at 15% and then stops.

[webmaster]

Cloudron version 9.0.13

Dec 05 11:07:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:13 box:shell mounts: mountpoint -q -- /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:21 box:user verify: peter.admin matched app password
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/box
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/mail
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/backups
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/oscarmagic
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/CoDeskPLMB
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/main-webmail-service
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/3b9f41b1-878c-4d52-a17a-d1e75b9e42b4/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/55c8ed23-a282-4a2d-ac2f-a62bbf328db1/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/magicwand
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/crm
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/contracts
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/cafe-platform-v1
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/bf455b80-5e6c-48a1-9c69-0046b82b0378/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/openUIOfficeNest
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/sportcafe
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/box
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/mail
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/backups
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/oscarmagic
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/CoDeskPLMB
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/main-webmail-service
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/3b9f41b1-878c-4d52-a17a-d1e75b9e42b4/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/55c8ed23-a282-4a2d-ac2f-a62bbf328db1/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/magicwand
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/crm
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/contracts
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/cafe-platform-v1
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/bf455b80-5e6c-48a1-9c69-0046b82b0378/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/openUIOfficeNest
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/sportcafe
Dec 05 11:07:35 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:35 box:asynctask start: FileSystemUsageTask(/dev/vdb1) started
Dec 05 11:07:35 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/hdparm.sh /dev/vdb1
Dec 05 11:07:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:48 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:57 box:logs tail: failed to kill children kill-child: target PID 143466 does not exist.
Dec 05 11:07:57
Dec 05 11:08:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:21 box:user verify: peter.admin matched app password
Dec 05 11:08:25 box:logs tail: failed to kill children kill-child: target PID 143626 does not exist.
Dec 05 11:08:25
Dec 05 11:08:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:21 box:user verify: peter.admin matched app password
Dec 05 11:09:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive

[james]

Hello @webmaster

Thanks for that.

From the log I assume you are using /mnt/blockstorage for Cloudron Volumes, is that correct?

[webmaster]

@james Yes it was all fine for months. This began all of a sudden.

Vultr block storage.
Here's some more that just appeared:

Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:35 box:asynctask start: FileSystemUsageTask(/dev/vdb1) started
Dec 05 11:07:35 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/hdparm.sh /dev/vdb1
Dec 05 11:07:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:48 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:57 box:logs tail: failed to kill children kill-child: target PID 143466 does not exist.
Dec 05 11:07:57
Dec 05 11:08:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:21 box:user verify: peter.admin matched app password
Dec 05 11:08:25 box:logs tail: failed to kill children kill-child: target PID 143626 does not exist.
Dec 05 11:08:25
Dec 05 11:08:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:21 box:user verify: peter.admin matched app password
Dec 05 11:09:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:53 box:asynctask stop: FileSystemUsageTask(/dev/vdb1) . sending abort signal
Dec 05 11:09:53 box:shell system: aborting 143440
Dec 05 11:09:53 box:shell system: aborting 143453
Dec 05 11:09:53 box:shell system: failed to kill children kill-child: target PID 143440 does not exist.
Dec 05 11:09:53
Dec 05 11:09:53 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef errored BoxError: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 05 11:09:53 at ChildProcess.emit (node:events:519:28)
Dec 05 11:09:53 at maybeClose (node:internal/child_process:1101:16)
Dec 05 11:09:53 at ChildProcess._handle.onexit (node:internal/child_process:304:5)
Dec 05 11:09:53 at Process.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 05 11:09:53 reason: 'Shell Error',
Dec 05 11:09:53 details: {},
Dec 05 11:09:53 stdout: '',
Dec 05 11:09:53 stdoutLineCount: 0,
Dec 05 11:09:53 stderr: 'Terminated\n',
Dec 05 11:09:53 stderrLineCount: 1,
Dec 05 11:09:53 code: 143,
Dec 05 11:09:53 signal: null,
Dec 05 11:09:53 timedOut: false,
Dec 05 11:09:53 terminated: true
Dec 05 11:09:53 }
Dec 05 11:09:53 box:system du error /mnt/volumes/ef20938604d944ee99f83a2791d89bef: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 box:asynctask start: FileSystemUsageTask(/dev/vdb1) finished
Dec 05 11:09:56 box:shell system: terminated 143453 kill-child: kill PID 143453 (command: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef)
Dec 05 11:09:56 kill-child: killing 143455
Dec 05 11:09:56 kill-child: killing 143454
Dec 05 11:09:56 kill-child: killing 143453

[james]

Hello @webmaster

Thanks, this log is very helpful.
Especially this part:

Dec 05 11:09:53 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef errored BoxError: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 05 11:09:53 at ChildProcess.emit (node:events:519:28)
Dec 05 11:09:53 at maybeClose (node:internal/child_process:1101:16)
Dec 05 11:09:53 at ChildProcess._handle.onexit (node:internal/child_process:304:5)
Dec 05 11:09:53 at Process.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 05 11:09:53 reason: 'Shell Error',
Dec 05 11:09:53 details: {},
Dec 05 11:09:53 stdout: '',
Dec 05 11:09:53 stdoutLineCount: 0,
Dec 05 11:09:53 stderr: 'Terminated\n',
Dec 05 11:09:53 stderrLineCount: 1,
Dec 05 11:09:53 code: 143,
Dec 05 11:09:53 signal: null,
Dec 05 11:09:53 timedOut: false,
Dec 05 11:09:53 terminated: true
Dec 05 11:09:53 }
Dec 05 11:09:53 box:system du error /mnt/volumes/ef20938604d944ee99f83a2791d89bef: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 box:asynctask start: FileSystemUsageTask(/dev/vdb1) finished
Dec 05 11:09:56 box:shell system: terminated 143453 kill-child: kill PID 143453 (command: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef)

Can you please run the following command on your server and share the output:

/usr/bin/sudo --non-interactive du -DsB1 /mnt/volumes/ef20938604d944ee99f83a2791d89bef

[webmaster]

@james Wondering if this is a Vultr degradation....

root@host:~# /usr/bin/sudo --non-interactive du -DsB1 /mnt/volumes/ef20938604d944ee99f83a2791d89bef
339621523456    /mnt/volumes/ef20938604d944ee99f83a2791d89bef
root@host:~#

[nebulon]

Since the task got killed, as indicated in the logs, while running it manually seems to work fine. This might be a memory issue then. If you can reproduce it, can you trigger it again to let it crash and then check systemctl --system to see the reason for it to get killed by the system? I suspect this is due to out-of-memory (oom).

[webmaster]

Doubt it... it never reaches more than 8gb and i have a 32gb server.

[webmaster]

We can't risk our site/business through cloudron - there's too many issues and zero direct support. Forums are not the right way to get support for a business critical app, especially when we pay you all this money every month. We will be cancelling our subscription. Cancel this request.

[nebulon]

Are we actually talking about the disk usage graph calculation or some app issue which causes downtime somehow?

For the memory issue, since Cloudron components may be artificially limited to cause less side-effects if an app is faulty, the limit could be reached and a process thus get killed by the kernel, despite having free memory on the overall system. Thus would be good to see the output of the system logs via journalctl --system at around the time of the process ending.

[joseph]

@webmaster we do offer direct support for businesses. We have a business plan exactly for businesses such as yours. If you are on a business plan, you can contact us at support@cloudron.io and we respond in a day.