Cloudron Forum

Thank you. That was my issue. I've followed the instructions on that thread and then truncated the syslog files which were around 50GB in size.

I previously received " /dev/sda is at 100% capacity" and apps started crashing 5 days ago when i created a 100GB volume and moved 30GB of data (Nextcloud, pixelfed, etc) out of the server. Today that 30 GB has been consumed by "Everything else (Ubuntu, etc) 82.23 GB" which was previously around 50GB.

I'm not sure how to diagnose the issue. I've checked previous threads on similar issues and backups aren't stored locally so /var/backups had about 100MB of files.

Cloudron Ver: v8.3.1

@joseph , Running 'cloudron-support --recreate-containers' fixed the issue. Thank you.

Hi Joseph, I had 9GB of space on the server when this happened. I run it on linode so I restored a snapshot from 2 days ago, upgrade the VPS which now has 35GB spare space and 2GB spare ram and then upgraded the server. After 2 retries the upgrade still fails which is quiet disappointing.

[no timestamp]  cmd: 'docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=3c0e900572f8e6a0bf8ab8cac5c745343970e5bb28148ea6e12a17ac1838b5fdb5cd182b8f8bf07bcd30fa27ac4bc5f309d9b5feaa103ccaf21ff6bc210c58401c585a92749fe2a3f6e88a9e5f86fe749a79bc1f6a87d3b8437cc3fa9dd2520c73bc0ab28cae8f83e5ef09308eb38f84084fd672e421185a0df73293ae328597 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=49b4e809bd7b9acaf8698e3afd7def7c919388c26d121ea3d17adf968bfb46d2a4d0d88ecd15ea360defc16addf9d0048910b10e7eeeb9336dc5436fc876cc7c463d1aa37b5a534746325092572b7650a53680f3dd8b8eeae5a440391b5ec0c07414c85720a0977d82b1f364f12f43fc490a2b9383420ac7a6680ffa510951b7 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2 '
[no timestamp]  }
Sep 26 10:29:48 box:platform startInfra: Failed to start services. retry=false (attempt 0): startMysql errored with code 125 message Command failed: docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=3c0e900572f8e6a0bf8ab8cac5c745343970e5bb28148ea6e12a17ac1838b5fdb5cd182b8f8bf07bcd30fa27ac4bc5f309d9b5feaa103ccaf21ff6bc210c58401c585a92749fe2a3f6e88a9e5f86fe749a79bc1f6a87d3b8437cc3fa9dd2520c73bc0ab28cae8f83e5ef09308eb38f84084fd672e421185a0df73293ae328597 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=49b4e809bd7b9acaf8698e3afd7def7c919388c26d121ea3d17adf968bfb46d2a4d0d88ecd15ea360defc16addf9d0048910b10e7eeeb9336dc5436fc876cc7c463d1aa37b5a534746325092572b7650a53680f3dd8b8eeae5a440391b5ec0c07414c85720a0977d82b1f364f12f43fc490a2b9383420ac7a6680ffa510951b7 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
[no timestamp]  Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/yellowtent/platformdata/mysql" to rootfs at "/var/lib/mysql": mount /home/yellowtent/platformdata/mysql:/var/lib/mysql (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
[no timestamp] 
Sep 26 10:29:48 box:platform BoxError: startMysql errored with code 125 message Command failed: docker run --restart=always -d --name=mysql --hostname mysql --net cloudron --net-alias mysql --log-driver syslog --log-opt syslog-address=unix:///home/yellowtent/platformdata/logs/syslog.sock --log-opt syslog-format=rfc5424 --log-opt tag=mysql --ip 172.18.30.1 -e CLOUDRON_MYSQL_TOKEN=3c0e900572f8e6a0bf8ab8cac5c745343970e5bb28148ea6e12a17ac1838b5fdb5cd182b8f8bf07bcd30fa27ac4bc5f309d9b5feaa103ccaf21ff6bc210c58401c585a92749fe2a3f6e88a9e5f86fe749a79bc1f6a87d3b8437cc3fa9dd2520c73bc0ab28cae8f83e5ef09308eb38f84084fd672e421185a0df73293ae328597 -e CLOUDRON_MYSQL_ROOT_HOST=172.18.0.1 -e CLOUDRON_MYSQL_ROOT_PASSWORD=49b4e809bd7b9acaf8698e3afd7def7c919388c26d121ea3d17adf968bfb46d2a4d0d88ecd15ea360defc16addf9d0048910b10e7eeeb9336dc5436fc876cc7c463d1aa37b5a534746325092572b7650a53680f3dd8b8eeae5a440391b5ec0c07414c85720a0977d82b1f364f12f43fc490a2b9383420ac7a6680ffa510951b7 -v /home/yellowtent/platformdata/mysql:/var/lib/mysql --label isCloudronManaged=true --cap-add SYS_NICE --read-only -v /tmp -v /run registry.docker.com/cloudron/mysql:3.4.3@sha256:8934c5ddcd69f24740d9a38f0de2937e47240238f3b8f5c482862eeccc5a21d2
[no timestamp]  Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/yellowtent/platformdata/mysql" to rootfs at "/var/lib/mysql": mount /home/yellowtent/platformdata/mysql:/var/lib/mysql (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
[no timestamp] 
[no timestamp]  at /home/yellowtent/box/src/shell.js:73:23
[no timestamp]  at ChildProcess.exithandler (node:child_process:430:5)
[no timestamp]  at ChildProcess.emit (node:events:518:28)
[no timestamp]  at maybeClose (node:internal/child_process:1105:16)
[no timestamp]  at ChildProcess._handle.onexit (node:internal/child_process:305:5)
Sep 26 10:31:26 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:32:10 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:34:34 box:locker Lock unreleased infra_start
Sep 26 10:35:09 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:39:34 box:locker Lock unreleased infra_start
Sep 26 10:41:26 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:32:10 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:34:34 box:locker Lock unreleased infra_start
Sep 26 10:35:09 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }
Sep 26 10:39:34 box:locker Lock unreleased infra_start
Sep 26 10:41:26 box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
[no timestamp]  Unknown error: 400 <html>
Dec 31 23:43:52 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  status: 500,
[no timestamp]  internalError: BoxError: Unknown error: 400 <html>
[no timestamp]  <head><title>400 Bad Request</title></head>
[no timestamp]  <body>
[no timestamp]  <center><h1>400 Bad Request</h1></center>
[no timestamp]  <hr><center>nginx</center>
[no timestamp]  </body>
[no timestamp]  </html>
[no timestamp] 
[no timestamp]  at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
[no timestamp]  at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
[no timestamp]  reason: 'External Error',
[no timestamp]  details: {}
[no timestamp]  },
[no timestamp]  details: {}
[no timestamp]  }

My Cloudron instance updated last night to 8.0.6 and it's been stuck on 'starting services'. I have tried rebooting and also restarting box service as mentioned on another post with no luck. The following is the output of the box log:

docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/yellowtent/platformdata/mysql" to rootfs at "/var/lib/mysql": mount /home/yellowtent/platformdata/mysql:/var/lib/mysql (via /proc/self/fd/6), flags: 0x5000: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.

    at /home/yellowtent/box/src/shell.js:73:23
    at ChildProcess.exithandler (node:child_process:430:5)
    at ChildProcess.emit (node:events:518:28)
    at maybeClose (node:internal/child_process:1105:16)
    at ChildProcess._handle.onexit (node:internal/child_process:305:5)
2024-09-24T21:52:34.615Z box:shell getSwaps execArgs: swapon ["--noheadings","--raw","--bytes","--show=type,size,used,name"]
HttpError: Unknown error: 400 <html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

    at getSystemGraphs (/home/yellowtent/box/src/routes/system.js:126:28)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  status: 500,
  internalError: BoxError: Unknown error: 400 <html>
  <head><title>400 Bad Request</title></head>
  <body>
  <center><h1>400 Bad Request</h1></center>
  <hr><center>nginx</center>
  </body>
  </html>
  
      at Object.getSystem (/home/yellowtent/box/src/graphs.js:106:46)
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
    reason: 'External Error',
    details: {}
  },
  details: {}
}