Filemanager broken after running out of disk space

[m3pilot]

After my Cloudron ran out of memory, I had to restore it from a backup which worked fine. The only problems I'm having now is that the filemanager does not work and I get and error that says it cannot connect to the filemanager server.

Here's the error from the logs.

GET /api/v1/apps/565f9baf-2ed1-4d79-b75c-d12fe409ce84/files/ 424 Failed Dependency Unable to connect to filemanager server 11.989 ms - 91

Thanks

[girish]

@m3pilot I think this is because the sftp container (also which servers up files for filemanager) is not running. Can you check the logs of the sftp service under Services ?

I suspect it's complaining about some existing key or could not generate key. This happens when the server runs out of disk space and it ends up with 0 length key files. If this is the case, you can try this:

rm /home/yellowtent/platformdata/sftp/ssh/ssh_host_rsa_key
rm /home/yellowtent/platformdata/sftp/ssh/ssh_host_rsa_key.pub 
docker restart sftp

Let me know if the service is green after the above. Otherwise, can you post the logs of the sftp container?

[m3pilot]

@girish

Thanks for the help. The sftp service has been showing green the entire time. I even tried to restart it but that did not help.

I removed the key files and restarted sftp from the command line. That didn't work either. The log shows that sftp tries to start but immediately crashes and it tries several times before giving up.

Here's a bunch of the sftp service log.

SFTP
Apr 24 17:04:14 Node.js v18.12.1
Apr 24 17:04:14 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:14 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:14 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:14 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:14 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:14 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:14 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:14 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:14 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:14 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:15 2023-04-24 22:04:15,727 INFO success: filemanager entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 24 17:04:15 2023-04-24 22:04:15,728 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:16 /app/code/filemanager/node_modules/bl/bl.js:36
Apr 24 17:04:16 2023-04-24 22:04:16,731 INFO spawned: 'filemanager' with pid 230
Apr 24 17:04:16 Node.js v18.12.1
Apr 24 17:04:16 TypeError: inherits is not a function
Apr 24 17:04:16 ^
Apr 24 17:04:16 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:16 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:16 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:16 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:16 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:16 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:16 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:16 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:16 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:16 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:16 inherits(BufferListStream, DuplexStream)
Apr 24 17:04:17 2023-04-24 22:04:17,885 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:17 2023-04-24 22:04:17,885 INFO success: filemanager entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 24 17:04:18 2023-04-24 22:04:18,888 INFO spawned: 'filemanager' with pid 237
Apr 24 17:04:19 /app/code/filemanager/node_modules/bl/bl.js:36
Apr 24 17:04:19 2023-04-24 22:04:19,021 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:19 Node.js v18.12.1
Apr 24 17:04:19 TypeError: inherits is not a function
Apr 24 17:04:19 ^
Apr 24 17:04:19 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:19 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:19 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:19 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:19 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:19 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:19 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:19 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:19 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:19 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:19 inherits(BufferListStream, DuplexStream)
/app/code/filemanager/node_modules/bl/bl.js:36
Apr 24 17:04:20 2023-04-24 22:04:20,025 INFO spawned: 'filemanager' with pid 244
Apr 24 17:04:20 2023-04-24 22:04:20,218 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:20 Node.js v18.12.1
Apr 24 17:04:20 TypeError: inherits is not a function
Apr 24 17:04:20 ^
Apr 24 17:04:20 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:20 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:20 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:20 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:20 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:20 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:20 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:20 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:20 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:20 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:20 inherits(BufferListStream, DuplexStream)
Apr 24 17:04:22 /app/code/filemanager/node_modules/bl/bl.js:36
Apr 24 17:04:22 2023-04-24 22:04:22,222 INFO spawned: 'filemanager' with pid 251
Apr 24 17:04:22 2023-04-24 22:04:22,353 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:22 Node.js v18.12.1
Apr 24 17:04:22 TypeError: inherits is not a function
Apr 24 17:04:22 ^
Apr 24 17:04:22 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:22 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:22 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:22 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:22 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:22 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:22 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:22 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:22 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:22 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:22 inherits(BufferListStream, DuplexStream)
Apr 24 17:04:25 /app/code/filemanager/node_modules/bl/bl.js:36
Apr 24 17:04:25 2023-04-24 22:04:25,358 INFO spawned: 'filemanager' with pid 258
Apr 24 17:04:25 2023-04-24 22:04:25,525 INFO exited: filemanager (exit status 1; not expected)
Apr 24 17:04:25 Node.js v18.12.1
Apr 24 17:04:25 TypeError: inherits is not a function
Apr 24 17:04:25 ^
Apr 24 17:04:25 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:25 at Module._compile (node:internal/modules/cjs/loader:1159:14)
Apr 24 17:04:25 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:25 at Module._extensions..js (node:internal/modules/cjs/loader:1213:10)
Apr 24 17:04:25 at Module._load (node:internal/modules/cjs/loader:878:12)
Apr 24 17:04:25 at Module.load (node:internal/modules/cjs/loader:1037:32)
Apr 24 17:04:25 at Module.require (node:internal/modules/cjs/loader:1061:19)
Apr 24 17:04:25 at Object.<anonymous> (/app/code/filemanager/node_modules/bl/bl.js:36:1)
Apr 24 17:04:25 at Object.<anonymous> (/app/code/filemanager/node_modules/tar-stream/extract.js:2:10)
Apr 24 17:04:25 at require (node:internal/modules/cjs/helpers:103:18)
Apr 24 17:04:25 inherits(BufferListStream, DuplexStream)
Apr 24 17:04:26 2023-04-24 22:04:26,526 INFO gave up: filemanager entered FATAL state, too many start retries too quickly

[girish]

@m3pilot not sure what's happening here, something seems totally wrong (like some mismatched node version). Are you able to contact us on support@cloudron.io to debug this further?