Crash when updating to v4.61.0

[rosano]

@girish said in Crash when updating to v4.61.0:

@rosano this seems something in Cloudron level and not at the ghost level. Can you check the usual suspects - disk space, backup configuration being correct etc ?

Not sure if this disk space should be an issue:

I looked at the logs and see some error:

Oct 08 11:09:42box:docker pullImage: {"status":"Extracting","progressDetail":{"current":245113694,"total":245113694},"progress":"[==================================================>] 245.1MB/245.1MB","id":"bf58462768b1"}
Oct 08 11:09:44box:docker pullImage: {"errorDetail":{"message":"failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/es-abstract/2020/Day.js: no space left on device"},"error":"failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/es-abstract/2020/Day.js: no space left on device"}
Oct 08 11:09:44box:docker pullImage error cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d: failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/es-abstract/2020/Day.js: no space left on device
Oct 08 11:09:44box:docker downloaded image cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d . error: true
layerError.includes is not a function
at IncomingMessage.<anonymous> (/home/yellowtent/box/src/docker.js:138:44)
at IncomingMessage.emit (node:events:525:35)
at IncomingMessage.emit (node:domain:489:12)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

I didn't notice that it was like this for over a month:

Also, are you on ubuntu 18.04 by any chance? If so, this is most likely the node issue.

Ubuntu 22.04.2

Any other hints?

[girish]

@rosano the server is running out of disk space to pull down the image. there is a bug in the error reporting logic which I have fixed now. But the main issue is that there is no disk space to download the latest ghost image.

[rosano]

This post is deleted!

[rosano]

@girish said in Crash when updating to v4.61.0:

@rosano the server is running out of disk space to pull down the image. there is a bug in the error reporting logic which I have fixed now. But the main issue is that there is no disk space to download the latest ghost image.

Seems like there's 4gb of swap and I would like to try without it since my memory is higher than before. But I'm not sure how to 'undo' the swap instructions. Would you mind sharing them here or adding to the docs page?

[girish]

@rosano you can swapoff /apps.swap and then resize that file as needed. Or delete that file and follow the instructions again.

[rosano]

Found some storage reduction ideas in Cannot update due to one tiny app filling up 20GB drive (FWIW my server storage capacity is 40gb)…

I tried running docker images:

REPOSITORY                                TAG                         IMAGE ID       CREATED        SIZE
cloudron/io.cloudron.surfer               20230926-081905-837cdc677   4b7368c3a664   2 weeks ago    2.59GB
cloudron/io.directus9.cloudronapp         20230926-074607-17122a91d   343da0deaf6b   2 weeks ago    2.88GB
cloudron/org.ghost.cloudronapp2           20230826-041735-9615c4784   113189f57639   6 weeks ago    3.37GB
registry.docker.com/cloudron/mail         3.9.3                       add54ace2ac9   7 weeks ago    3.42GB
registry.docker.com/cloudron/turn         1.6.0                       0e227fff89a8   2 months ago   2.5GB
registry.docker.com/cloudron/sftp         3.7.4                       f3c7819397ff   2 months ago   2.52GB
cloudron/org.tt_rss.cloudronapp           20230707-145335-4132ad578   2ca269b57f3d   3 months ago   2.55GB
registry.docker.com/cloudron/postgresql   5.0.7                       dae98c0f8515   3 months ago   2.86GB
cloudron/io.n8n.cloudronapp               20230622-090234-1238ce799   512350c52d46   3 months ago   3.46GB
cloudron/org.ghost.cloudronapp2           20230520-114757-246f8325c   1f0c61fa6522   4 months ago   3.38GB
registry.docker.com/cloudron/redis        3.5.0                       d2d2522779b5   5 months ago   2.51GB
cloudron/github.pages.cloudronapp         20230429-170050-5438eb21a   4f0d39c9edd2   5 months ago   2.61GB
registry.docker.com/cloudron/mongodb      4.3.7                       da59610c351b   6 months ago   2.78GB
registry.docker.com/cloudron/mysql        3.3.7                       16c290018ffa   6 months ago   2.82GB
cloudron/org.yourls.cloudronapp           20230311-140358-906f40ae4   661b7706aff9   7 months ago   2.5GB
registry.docker.com/cloudron/graphite     3.3.0                       1de043e7f9a8   7 months ago   2.57GB
cloudron/org.radicale.cloudronapp2        20230105-192313-34404fcdb   3a7f634fa51f   9 months ago   2.51GB
cloudron/org.ghost.cloudronapp2           20230102-095442-905759029   d1cfc01c9cf1   9 months ago   3.49GB

Curious why each app seems to have gigabytes in size, but I don't know what any of this means.

Tried docker image prune -a:

Deleted Images:
untagged: registry.docker.com/cloudron/base:4.0.0
untagged: registry.docker.com/cloudron/base@sha256:31b195ed0662bdb06a6e8a5ddbedb6f191ce92e8bee04c03fb02dd4e9d0286df
deleted: sha256:ed5b67be5b16b1fa810834a638b62f7d4b1b5ac699f5d46347d08ccbd591a0be

Total reclaimed space: 0B

@girish I tried sudo swapoff -a:

swapoff: /apps.swap: swapoff failed: Cannot allocate memory

Tried sudo rm /apps.swap:

rm: cannot remove '/apps.swap': Operation not permitted

I don't have much experience to understand what to do here so if you have further hints I would appreciate it

[rosano]

@girish actually trying sudo swapoff /apps.swap seems to have worked and I can sudo rm /apps.swap now. 4gb free

As an aside, if I truncate -s 1G /apps.swap and then either sudo swapon -a or sudo swapon /apps.swap I get this error:

swapon: /apps.swap: swapon failed: Invalid argument

I guess will leave it off for now, but maybe at some point I would try to turn it on.

[rosano]

@girish Okay still getting this error with 4.67 GB free:

Oct 11 09:31:57box:docker pullImage: {"errorDetail":{"message":"failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/express-hbs/test/apps/async/index.js: no space left on device"},"error":"failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/express-hbs/test/apps/async/index.js: no space left on device"}
Oct 11 09:31:57box:docker pullImage error cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d: failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/express-hbs/test/apps/async/index.js: no space left on device
Oct 11 09:31:57box:docker downloaded image cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d . error: true
layerError.includes is not a function
at IncomingMessage.<anonymous> (/home/yellowtent/box/src/docker.js:138:44)
at IncomingMessage.emit (node:events:525:35)
at IncomingMessage.emit (node:domain:489:12)
at endReadableNT (node:internal/streams/readable:1359:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

Seems like the same issue as the other thread I referenced, but not sure what else I can do to reduce space if that's the issue. Any ideas?

[girish]

@rosano can you can try docker pull cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d directly. Is your server so short of space? How much is remaining?

[girish]

If you run just swapon, it will say the swap devices also. But really, trying to shave off 2-3GB here and there is just going to error somewhere else (or sometime later), I think.

[rosano]

@girish said in Crash when updating to v4.61.0:

@rosano can you can try docker pull cloudron/org.ghost.cloudronapp2:20230904-084557-30794834d directly.

Still a space issue:

failed to register layer: Error processing tar file(exit status 1): open /home/cloudron/ghost/versions/5.61.1/node_modules/node-gyp/gyp/pylib/gyp/common_test.py: no space left on device

Turns out swap comes back after rebooting, so I deleted it again but still get the same issue.

Is your server so short of space? How much is remaining?

According to df -h 5-6 GB after deleting swap?:

/dev/sda1        38G   30G  5.8G  84% /

It surprises me more than this would be necessary for Ghost. I wonder what's going on.

[girish]

@rosano indeed, that should be ample. maybe try a server reboot to see if some process has open files preventing from cleanup . Before you reboot, you might have to comment out the swap mount line in /etc/fstab.

[rosano]

Ran docker volume prune to reclaim 1.3 GB. Was able to successfully pull directly with 7.7 GB free. I ran docker image prune -a to remove that pull and was able to install via the Cloudron panel.

Will comment out the swap but hopefully there isn't some other issue as it's odd to need so much space in order to update ghost.

[girish]

@rosano there's an intermediate transition going on for the docker images (we are updating the docker image to version 4.2.0). Once most apps are updated, you should be able to reclaim more space.

[rosano]

Sorry to extend this thread, just noticing that the swap seems to return after rebooting even though /apps.swap none swap sw 0 0 is commented out in /etc/fstab. This blocks other attemps when Ghost updates… For now I'll just delete swap it occurs, but if there's another step please share.

[girish]

@rosano instead of deleting /apps.swap , just keep it 0 size and remove the fstab line. Cloudron recreates the swap file on reboot (since that is the recommended setup for linux). so truncate -s0 /apps.swap is good enough.