Disk not being calculated, just stuck at 15% and then stops.

[webmaster]

Disk not being calculated, just stuck at 15% and then stops.

I've rebooted the block storage and the server - no diff.

Any thoughts how to recalculate, maybe via ssh?

[james]

Hello @webmaster and welcome to the Cloudron forum

Unfortunately you have decided to ignore the topic template which would have speed up the analysis of your reported issue.

Please share the following details:

• Cloudron Version
• When this issue occurs, the log file /home/yellowtent/platformdata/logs/box.log

I was not able to reproduce this issue on Cloudron 9.0.13

[webmaster]

Cloudron version 9.0.13

Dec 05 11:07:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:13 box:shell mounts: mountpoint -q -- /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:21 box:user verify: peter.admin matched app password
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/box
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/mail
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/backups
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/oscarmagic
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/CoDeskPLMB
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/main-webmail-service
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/3b9f41b1-878c-4d52-a17a-d1e75b9e42b4/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/55c8ed23-a282-4a2d-ac2f-a62bbf328db1/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/magicwand
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/crm
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/contracts
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/cafe-platform-v1
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/bf455b80-5e6c-48a1-9c69-0046b82b0378/data
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/openUIOfficeNest
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/sportcafe
Dec 05 11:07:29 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:29 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/platformdata
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/box
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/boxdata/mail
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/backups
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /var/lib/docker
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/oscarmagic
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/CoDeskPLMB
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/main-webmail-service
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/3b9f41b1-878c-4d52-a17a-d1e75b9e42b4/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/55c8ed23-a282-4a2d-ac2f-a62bbf328db1/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/magicwand
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/crm
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/contracts
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/cafe-platform-v1
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /home/yellowtent/appsdata/bf455b80-5e6c-48a1-9c69-0046b82b0378/data
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/openUIOfficeNest
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /mnt/volumes/ef20938604d944ee99f83a2791d89bef/sportcafe
Dec 05 11:07:35 box:shell system: swapon --noheadings --raw --bytes --show=type,size,used,name
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:35 box:asynctask start: FileSystemUsageTask(/dev/vdb1) started
Dec 05 11:07:35 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/hdparm.sh /dev/vdb1
Dec 05 11:07:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:48 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:57 box:logs tail: failed to kill children kill-child: target PID 143466 does not exist.
Dec 05 11:07:57
Dec 05 11:08:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:21 box:user verify: peter.admin matched app password
Dec 05 11:08:25 box:logs tail: failed to kill children kill-child: target PID 143626 does not exist.
Dec 05 11:08:25
Dec 05 11:08:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:21 box:user verify: peter.admin matched app password
Dec 05 11:09:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive

[james]

Hello @webmaster

Thanks for that.

From the log I assume you are using /mnt/blockstorage for Cloudron Volumes, is that correct?

[webmaster]

@james Yes it was all fine for months. This began all of a sudden.

Vultr block storage.
Here's some more that just appeared:

Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /apps.swap
Dec 05 11:07:35 box:shell df: df -B1 --output=source,fstype,size,used,avail,pcent,target /swapfile
Dec 05 11:07:35 box:asynctask start: FileSystemUsageTask(/dev/vdb1) started
Dec 05 11:07:35 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/hdparm.sh /dev/vdb1
Dec 05 11:07:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:48 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef
Dec 05 11:07:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:07:57 box:logs tail: failed to kill children kill-child: target PID 143466 does not exist.
Dec 05 11:07:57
Dec 05 11:08:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:21 box:user verify: peter.admin matched app password
Dec 05 11:08:25 box:logs tail: failed to kill children kill-child: target PID 143626 does not exist.
Dec 05 11:08:25
Dec 05 11:08:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:08:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:00 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:10 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:20 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:21 box:user verify: peter.admin matched app password
Dec 05 11:09:30 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:40 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:50 box:apphealthmonitor app health: 8 running / 4 stopped / 0 unresponsive
Dec 05 11:09:53 box:asynctask stop: FileSystemUsageTask(/dev/vdb1) . sending abort signal
Dec 05 11:09:53 box:shell system: aborting 143440
Dec 05 11:09:53 box:shell system: aborting 143453
Dec 05 11:09:53 box:shell system: failed to kill children kill-child: target PID 143440 does not exist.
Dec 05 11:09:53
Dec 05 11:09:53 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef errored BoxError: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 05 11:09:53 at ChildProcess.emit (node:events:519:28)
Dec 05 11:09:53 at maybeClose (node:internal/child_process:1101:16)
Dec 05 11:09:53 at ChildProcess._handle.onexit (node:internal/child_process:304:5)
Dec 05 11:09:53 at Process.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 05 11:09:53 reason: 'Shell Error',
Dec 05 11:09:53 details: {},
Dec 05 11:09:53 stdout: '',
Dec 05 11:09:53 stdoutLineCount: 0,
Dec 05 11:09:53 stderr: 'Terminated\n',
Dec 05 11:09:53 stderrLineCount: 1,
Dec 05 11:09:53 code: 143,
Dec 05 11:09:53 signal: null,
Dec 05 11:09:53 timedOut: false,
Dec 05 11:09:53 terminated: true
Dec 05 11:09:53 }
Dec 05 11:09:53 box:system du error /mnt/volumes/ef20938604d944ee99f83a2791d89bef: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 box:asynctask start: FileSystemUsageTask(/dev/vdb1) finished
Dec 05 11:09:56 box:shell system: terminated 143453 kill-child: kill PID 143453 (command: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef)
Dec 05 11:09:56 kill-child: killing 143455
Dec 05 11:09:56 kill-child: killing 143454
Dec 05 11:09:56 kill-child: killing 143453

[james]

Hello @webmaster

Thanks, this log is very helpful.
Especially this part:

Dec 05 11:09:53 box:shell system: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef errored BoxError: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:82:23)
Dec 05 11:09:53 at ChildProcess.emit (node:events:519:28)
Dec 05 11:09:53 at maybeClose (node:internal/child_process:1101:16)
Dec 05 11:09:53 at ChildProcess._handle.onexit (node:internal/child_process:304:5)
Dec 05 11:09:53 at Process.callbackTrampoline (node:internal/async_hooks:130:17) {
Dec 05 11:09:53 reason: 'Shell Error',
Dec 05 11:09:53 details: {},
Dec 05 11:09:53 stdout: '',
Dec 05 11:09:53 stdoutLineCount: 0,
Dec 05 11:09:53 stderr: 'Terminated\n',
Dec 05 11:09:53 stderrLineCount: 1,
Dec 05 11:09:53 code: 143,
Dec 05 11:09:53 signal: null,
Dec 05 11:09:53 timedOut: false,
Dec 05 11:09:53 terminated: true
Dec 05 11:09:53 }
Dec 05 11:09:53 box:system du error /mnt/volumes/ef20938604d944ee99f83a2791d89bef: /usr/bin/sudo exited with code 143 signal null
Dec 05 11:09:53 box:asynctask start: FileSystemUsageTask(/dev/vdb1) finished
Dec 05 11:09:56 box:shell system: terminated 143453 kill-child: kill PID 143453 (command: /usr/bin/sudo --non-interactive /home/yellowtent/box/src/scripts/du.sh /mnt/volumes/ef20938604d944ee99f83a2791d89bef)

Can you please run the following command on your server and share the output:

/usr/bin/sudo --non-interactive du -DsB1 /mnt/volumes/ef20938604d944ee99f83a2791d89bef

[webmaster]

@james Wondering if this is a Vultr degradation....

root@host:~# /usr/bin/sudo --non-interactive du -DsB1 /mnt/volumes/ef20938604d944ee99f83a2791d89bef
339621523456    /mnt/volumes/ef20938604d944ee99f83a2791d89bef
root@host:~#

[nebulon]

Since the task got killed, as indicated in the logs, while running it manually seems to work fine. This might be a memory issue then. If you can reproduce it, can you trigger it again to let it crash and then check systemctl --system to see the reason for it to get killed by the system? I suspect this is due to out-of-memory (oom).

[webmaster]

Doubt it... it never reaches more than 8gb and i have a 32gb server.