@jdaviescoates NetCup is 100% green https://www.netcup.eu/ueber-netcup/oekostrom.php

Hi all,

I was looking into storage usage of my Cloudron instance and I stumbled across some unusually high storage usage of my backups.. To clarify, my Cloudron server runs:

• a NextCloud instance with about 16GB worth of files (Cloudron says it's 40GB, but I guess there's some footprint for the installation and file versioning etc.)
• a Wordpress instance of a few MBs (Cloudron says 60MB)

Now, as far as I understand backups in rsync start with a snapshot and work incrementally from there. I do not upload/remove too many files from my server so the storage remains fairly constant. Therefore, I'd expect a backup size somewhere not too far above the 40GB.

I checked my backups and they were covering a quite whopping 66GB. Taking a closer look revealed some tar.gz files and my hunch is that they are remnants of my old backup setup into tarballs. My snapshot folder is filled as such:

• /app_<NextCloud-app-ID> (40GB)
• app_<NextCloud-app-ID>.tar.gz (20GB)
• /app_<WordPress-app-ID> (30MB)
• /box
• /box.tar.gz
• app_<unknown-app-ID>.tar.gz (I think this might be an old app I had)

So the tarballs are together over 20GB in size, confirming my hunch (because it would make my backups 46GB in size).

TLDR; Are tarballs saved in the backup/snapshot folder when backups are in rsync? Or can I remove tarballs in my backup/snapshot folder?

@jdaviescoates I use them for my own Cloudron instance! Works like a charm, they even have 18.04 with preinstalled Cloudron images so it was zero effort for setup I run my backups somewhere else though, so not sure about those.

Thanks for your reply @girish. I contacted customer support of Scaleway and they told me they do not throttle their object storages (which is what I backup to).

Scanning the log files, I found that it did in fact crash on a specific series of files. I think the root cause for this was that there was a single device of mine that halted on an auto-upload via the NextCloud app. I rarely use this device and did not notice it was stuck on an upload for days. I removed the files and ran another backup, this time it did not crash.

Is it possible that the backup script can timeout on such a partial/corrupted file?

@jdaviescoates My backups go via Scaleway. Was easy to setup because there was a Cloudron preset and guide for it, and it's free up to 75GBs AFAIK not green though..

@fbartels Indeed! I changed the title accordingly.

@girish They are unusual I would say, yes. It's a file that is being updated on a daily basis so it has a file history of 30 different versions in NextCloud. They are .db.crypt12 files, which is a database backup of WhatsApp about 120MB in size.

Thanks for your reply @girish. I contacted customer support of Scaleway and they told me they do not throttle their object storages (which is what I backup to).

Scanning the log files, I found that it did in fact crash on a specific series of files. I think the root cause for this was that there was a single device of mine that halted on an auto-upload via the NextCloud app. I rarely use this device and did not notice it was stuck on an upload for days. I removed the files and ran another backup, this time it did not crash.

Is it possible that the backup script can timeout on such a partial/corrupted file?

Hi all,

I've been using a Scaleway storage to backup my instance on a 3-day basis but I've been getting various timeout errors since the last 5 days (each day):

Message: write EPIPE HTTP Code: NetworkingError.

Message: Your socket connection to the server was not read from or written to within the timeout period. HTTP Code: RequestTimeout.

Message: Socket timed out without establishing a connection HTTP Code: TimeoutError.

Message: write EPIPE HTTP Code: NetworkingError.

The errors only happen after an hour or so in the backup process and files are added to my Scaleway instance. The error always pops up somewhere deep inside a gibberish directory (I think it's encrypted?) but never the same one so it seems. It's always my NextCloud instance and the directory always seems to match in some way, something like:

snapshot/app_xxxx/....1NRyroOTEk..../....YDrvyp7W5N..../..a../..b../..c../..d..
snapshot/app_xxxx/....1NRyroOTEk..../....YDrvyp7W5N..../..k../..l../..m../..n..

The only thing I can think of is that I messed up my NextCloud instance a bit because I removed a file directly in the terminal, which apparently causes the instance to get confused with its database. As recommended on other NextCloud posts that I found, I ran a command to reindex my database. Something like this:

sudo -u www-data php occ files:scan --all

I ran it within the Cloudron terminal of my NextCloud instance.

Any ideas on how to debug this problem? Is it related to the described hickup? Or is it related to Cloudron/Scaleway?

My backup instance started throwing errors since yesterday. I am not sure what's up here. Please find the error log below (I removed the encrypted names).

Backup failed: Error uploading snapshot/app_<xxxx>/<yyyy>. Message: The specified bucket does not exist. HTTP Code: NoSuchBucket. Logs are available here. Will be retried in 4 hours

This repeated three more times in a similar way but with a different file:

Backup failed: Error uploading snapshot/app_<xxyy>/<yyzz>/<yyaa>. Message: The specified bucket does not exist. HTTP Code: NoSuchBucket. Logs are available here. Will be retried in 4 hours

I thought I'd give it a day but it threw 4 more errors the next day. Different error though.

Backup failed: Error copying snapshot/app_<xxxx>/<yybb>/<yycc>/<yydd>/<yyee>/<yyff>/<yygg> (161696960 bytes): OperationAborted OperationAborted: A conflicting conditional operation is currently in progress against this resource. Please try again.. Logs are available here. Will be retried in 4 hours

I have a Scaleway bucket and found no files of yesterday's backups (4x NoBucket error). I did find four folders containing the four attempts that got aborted the day after (my bucket size exploded, lol..)

Any ideas on how to solve this?

@jdaviescoates My backups go via Scaleway. Was easy to setup because there was a Cloudron preset and guide for it, and it's free up to 75GBs AFAIK not green though..

@jdaviescoates I use them for my own Cloudron instance! Works like a charm, they even have 18.04 with preinstalled Cloudron images so it was zero effort for setup I run my backups somewhere else though, so not sure about those.

@jdaviescoates NetCup is 100% green https://www.netcup.eu/ueber-netcup/oekostrom.php

Hi all,

I was looking into storage usage of my Cloudron instance and I stumbled across some unusually high storage usage of my backups.. To clarify, my Cloudron server runs:

• a NextCloud instance with about 16GB worth of files (Cloudron says it's 40GB, but I guess there's some footprint for the installation and file versioning etc.)
• a Wordpress instance of a few MBs (Cloudron says 60MB)

Now, as far as I understand backups in rsync start with a snapshot and work incrementally from there. I do not upload/remove too many files from my server so the storage remains fairly constant. Therefore, I'd expect a backup size somewhere not too far above the 40GB.

I checked my backups and they were covering a quite whopping 66GB. Taking a closer look revealed some tar.gz files and my hunch is that they are remnants of my old backup setup into tarballs. My snapshot folder is filled as such:

• /app_<NextCloud-app-ID> (40GB)
• app_<NextCloud-app-ID>.tar.gz (20GB)
• /app_<WordPress-app-ID> (30MB)
• /box
• /box.tar.gz
• app_<unknown-app-ID>.tar.gz (I think this might be an old app I had)

So the tarballs are together over 20GB in size, confirming my hunch (because it would make my backups 46GB in size).

TLDR; Are tarballs saved in the backup/snapshot folder when backups are in rsync? Or can I remove tarballs in my backup/snapshot folder?

I was unaware of the incremental feature in rsync. That will work for me, thank you!

Hi there Cloudronians,

My server works like a charm but there is one feature that I would like to see. Currently, the amount of backups that I have of my server varies grealy, so the amount of storage space that I need on my backup drive varies equally much. I would like to have this more constant so the backup drive is not suddenly completely filled up when there are a couple of updates together with the usual backup schedule.

Would it be an option to limit the amount of backups? Say, maximum of 1 backup related to updates and 1 backup related to regular backup scheduling?