Cloudron System Backup Blocked By Prometheus

[BrutalBirdie]

@girish after your latest update for the prometheus app I came back to this issue. (thanks for the update again)

It seems like this issue still persists and it seems to be the 0 byte lock file in /app/data/runtime of the prometheus app.
This lock file is auto generated on every app restart, so having automated backups enabled with this app blocks the whole backup process of the system.

I just tested a normal backup of the single app with lock file in place and it got stuck pretty much immediate.
Then I removed the lock file and the single app backup worked.
I am now testing a full system backup with the lock file removed.
edit:
Yep full backup now working as well.

Did anyone of the @appdev had this issue as well if you use Prometheus with tgz backups?

If this issue persist and it can be replicated with literally every app restart I think a startup check for this file (and deletion) could solve this.

Will do some more testing and research.

[marcusquinn]

@eriktad You around to take a look?

[girish]

@brutalbirdie are you able to zip up the entire /app/data and send it to me on support@ ? I have tried with zero length files many times and it hasn't been a problem. Just tried again now as well with a fresh prometheus and lock file in place. Would be great to get this fixed.

I think it's some bug in the tar module we use and it seems to happen because of some specific directory structure and zero length file.

@marcusquinn it's not a prometheus bug, some backup code issue.

[BrutalBirdie]

@girish tar worked without a problem with the console.

Also send you the lag as file in chat.

https://paste.cloudron.dev/gofuzafiwu.sql

[BrutalBirdie]

Super strange.
I now tried a single app backup with the lock file in place and no problem.

I will now enable auto backup for this app again and see if the full system backup gets stuck again.

scratches had

[nebulon]

Is this still an issue or was it just some temporary fluke?

[Jan Macenka]

Unfortunately I am experiencing the same bug currently... having a backup of my metrics would be kind of important to me.

@BrutalBirdie any know work-a-rounds?

[BrutalBirdie]

@Jan-Macenka sadly no. I just excluded prometheus from the normal backup and do it manually.
So It does not block the full backup process.

[jdaviescoates]

@nebulon said in Cloudron System Backup Blocked By Prometheus:

Is this still an issue

Sounds like it.

[girish]

@Jan-Macenka yeah, there's some bug in the fs module we use . We are replacing it next release, so maybe this bug goes away.