Cloudron System Backup Blocked By Prometheus

[BrutalBirdie]

Cloudron Version: v6.1.2

The full system Backup is stuck at the prometheus app.

ID cfc4bfb0-4235-49dd-ae72-6ebeca56f788

Backup Tasks logs (to long for this post)

https://paste.cloudron.dev/imuvedujer.makefile

Size of /app/data/

root@cfc4bfb0-4235-49dd-ae72-6ebeca56f788:/app/code# cd /app/data/
root@cfc4bfb0-4235-49dd-ae72-6ebeca56f788:/app/data# du -h -d 1
1.2G    ./runtime
8.0K    ./config
1.2G    .
root@cfc4bfb0-4235-49dd-ae72-6ebeca56f788:/app/data# 

This is very odd.

Someone else had this issue?
Because I know on a second Cloudron instance there is no problem. (Also version 6.1.2)

[girish]

@brutalbirdie this seems to be like some bug in the backup system. I have seen this issue (rarely) where the tgz uploader will get stuck. Can you confirm you are using tgz backups? Does it work if you disable prometheus app from backup?

Generally, if you clean up / remove some files randomly inside prometheus it will start working. For example, can you check if there are 0 byte files in it's app data?

[BrutalBirdie]

@girish jep its tgz

@luckow check this out.

and yes there is a 0 byte file.

root@cfc4bfb0-4235-49dd-ae72-6ebeca56f788:/app/data# ls -lah runtime/
total 124K
drwxr-xr-x 26 cloudron cloudron 4.0K Mar  1 23:00 .
drwxr-xr-x  4 cloudron cloudron 4.0K Jan 20 09:55 ..
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 15 15:00 01EYK39BZERGGNNPD19A1Z4W0B
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 16 09:00 01EYN12X5W38W4ZTHSVPG8Y3SH
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 17 03:00 01EYPYWK4B43R2SMK2D0QEMVGE
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 17 21:00 01EYRWP37CS62GXD9ZVBHKDDBW
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 18 15:00 01EYTTFEB6VHK6EQ02PVFJ0P35
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 19 09:00 01EYWR90J58PZDR9RG7F9BRASJ
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 20 03:00 01EYYP2G8DGGWPHAPN04KZBWA4
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 20 21:00 01EZ0KW5BT8BEW6T89YXGB2KNV
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 21 15:00 01EZ2HNPKAW6D18QQQHJC73G82
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 22 09:00 01EZ4FFAK24KCKKKB4NZYY6NW3
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 23 03:00 01EZ6D8WHRCAG7KMY4V8B7GGY2
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 23 21:00 01EZ8B2B1FV13MF7CQCNJ76DTF
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 24 15:00 01EZA8WACAPQDYEMMA6DBHHPZC
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 25 09:00 01EZC6NC8T2DHBQ6MQX951K8RR
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 26 03:00 01EZE4F1TB8EAWRXX6RYDKD04G
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 26 21:00 01EZG28H0GCHD871FB95373FZF
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 27 15:00 01EZJ01ZSCQGZZDMCP6FZ93P8K
drwxr-xr-x  3 cloudron cloudron 4.0K Feb 28 11:00 01EZM4Q7SPACE1MYDZ70DFC5ND
drwxr-xr-x  3 cloudron cloudron 4.0K Mar  1 05:00 01EZP2GXSSPV9F52S6BNGNBR0C
drwxr-xr-x  3 cloudron cloudron 4.0K Mar  1 21:00 01EZQSEEAZT8HKGD9Q9YA4TKNP
drwxr-xr-x  3 cloudron cloudron 4.0K Mar  1 23:00 01EZR0A5KAHWE6YVRP9DWX7B3H
drwxr-xr-x  3 cloudron cloudron 4.0K Mar  1 23:00 01EZR0AGKZNAT2P7JH7FSBX9D5
drwxr-xr-x  2 cloudron cloudron 4.0K Mar  1 23:00 chunks_head
-rw-r--r--  1 cloudron cloudron    0 Jan 20 09:55 lock
-rw-r--r--  1 cloudron cloudron  20K Mar  1 13:30 queries.active
drwxr-xr-x  3 cloudron cloudron 4.0K Mar  1 23:00 wal

[BrutalBirdie]

@girish after your latest update for the prometheus app I came back to this issue. (thanks for the update again)

It seems like this issue still persists and it seems to be the 0 byte lock file in /app/data/runtime of the prometheus app.
This lock file is auto generated on every app restart, so having automated backups enabled with this app blocks the whole backup process of the system.

I just tested a normal backup of the single app with lock file in place and it got stuck pretty much immediate.
Then I removed the lock file and the single app backup worked.
I am now testing a full system backup with the lock file removed.
edit:
Yep full backup now working as well.

Did anyone of the @appdev had this issue as well if you use Prometheus with tgz backups?

If this issue persist and it can be replicated with literally every app restart I think a startup check for this file (and deletion) could solve this.

Will do some more testing and research.

[marcusquinn]

@eriktad You around to take a look?

[girish]

@brutalbirdie are you able to zip up the entire /app/data and send it to me on support@ ? I have tried with zero length files many times and it hasn't been a problem. Just tried again now as well with a fresh prometheus and lock file in place. Would be great to get this fixed.

I think it's some bug in the tar module we use and it seems to happen because of some specific directory structure and zero length file.

@marcusquinn it's not a prometheus bug, some backup code issue.

[BrutalBirdie]

@girish tar worked without a problem with the console.

Also send you the lag as file in chat.

https://paste.cloudron.dev/gofuzafiwu.sql

[BrutalBirdie]

Super strange.
I now tried a single app backup with the lock file in place and no problem.

I will now enable auto backup for this app again and see if the full system backup gets stuck again.

scratches had

[nebulon]

Is this still an issue or was it just some temporary fluke?

[Jan Macenka]

Unfortunately I am experiencing the same bug currently... having a backup of my metrics would be kind of important to me.

@BrutalBirdie any know work-a-rounds?

[BrutalBirdie]

@Jan-Macenka sadly no. I just excluded prometheus from the normal backup and do it manually.
So It does not block the full backup process.

[jdaviescoates]

@nebulon said in Cloudron System Backup Blocked By Prometheus:

Is this still an issue

Sounds like it.

[girish]

@Jan-Macenka yeah, there's some bug in the fs module we use . We are replacing it next release, so maybe this bug goes away.