Cloudron Forum

Hey all, writing up some notes from a recent migration. To me (as but an enthusiast) how to go about this was not obvious from the docs or existing posts, so hopefully this is helpful to others.

My setup for context:

• cloudron: a hetzner auction server with mixed HDD+SDD
• backup box: a storage VPS in a different geographical location, using SSHFS (encrypted rsync)
  • had a good experience with this so far, the ability to use hardlinks is a game changer for storage size, as well as not being charged per request
  • requires manually setting up a secure ssh box. I used unattended-upgrades and ufw
  • for peace of mind, i also do an offline manual backup to a drive at home every ~3-6 months
• Mixed HDD+SSD was useful for keeping cost low while balancing the needs of large apps like Nextcloud vs most other apps on cloudron. I haven't had much luck mounting external storage, so liked the idea of a bigger drive locally. the deal i found happened to have 3 drives and i laid them out like this
  • 2x512GB SSD with RAID1; default drive mounted at /, for apps where i care about both uptime & speed (email,websites..)
  • 1x1TB SSD; less critical apps where i care about speed, media-heavy apps like immich
  • 1x2TB HDD; large storage apps (Nextcloud)
  • I haven't yet set up email RAID monitoring if anyone has advice there

Before the migration i had to

• partition drives, mount extra SSD and HDD in /etc/fstab, set up RAID
• whitelist ip of the new server in my domain provider's API if needed. this was the case for namecheap.
• note down my ssh private key from the backup user (normally in /home/yellowtent/ssh) on the old server before powering it down. the backup config file doesn't include this automatically.
• i think it's best to harden SSH and firewall after fully completing restore. i tried to do this after installing cloudron but before restoring via the web interface, but seemed to always run into issues. not sure if other people have had this experience.

I ran into an issue where on my root 0.5TB drive, nextcloud would not fit. there wasn't a clear way to assign a data directory for it in advance of restore, so i had to restore it on a per-app basis via the following:

• start the restore via https://your-new-ip
• wait until the dashboard is up
• mount additional drives as volumes (i used filesystem(mountpoint)). I believe currently mounting them in /etc/fstab does not automatically make them recognized within cloudron.
• go to large app->uninstall menu->Archive
  • there didn't seem to be a way to pause the restore, or change data directory while running the restore, so this was the only way i found to prevent completely filling my drive. after a while when restore for other apps finished, this app was left in Error state and I could then successfully archive on the second attempt.
  • download app backup config from backups->archive
• install new instance of large app from the appstore that matches the backup app version
  • download a specific version of an app by changing the appstore URL: ie https://my.server.com/#/appstore/com.nextcloud.cloudronapp?version=5.5.2 <-- change to version=x.x.x
• move this app's data directory to the volume where your additional drive is mounted (app settings->storage->data directory)
• import app backup using the config file downloaded earlier

Curious how other's experience has been with large migrations, if you've found easier ways. Also your server/backup setups for dealing with apps like Nextcloud. I think some of this process can be improved or clarified in documentation/error messages. I.e. a note to remember to whitelist new server IP in domain provider API.

Having an option to mount volumes and assign per-app data-directory as part of the restore wizard would be fantastic.

I think it probably makes more sense for most people who are using a storage VPS to forego installing Cloudron+Minio and instead just connect to their VPS with SSHFS.

If you use Minio's S3 api, there's a big disadvantage of no hard-links with rsync right? So total backup storage balloons if you want anything but a barebones retention policy. In my case the difference was around 5TB . On the other hand setting up a secure SSH server is almost easier than a second cloudron (get a firewall going and auto update with unattended-upgrades). Not sure how people's experience is with backup stability over SSHFS these days.

Saw that ServaRica had some good prices competitive with Hetzner / alphaVPS.

Finally managed to get backups stable with backblaze for a ~1TB cloudron with these settings (using encryption and rsync):

Some notes:

• During initial backups, memory was by far the most common failure. The amount needed was higher than I first expected for a given upload size and concurrency. I found (upload part size) x (upload concurrency) x 4 was roughly the amount needed. i.e. (64mb)*(20)*4 = ~5120mb.
• Scheduling backups at a time when server load is low and allotting as much memory as comfortable helped stability a lot. I ended up keeping my memory limit much higher than strictly necessary. This could probably be reduced, but stability is more important to me at the moment.
• In the graph above memory tended to spike once per backup- @girish maybe this is an issue that could be optimized?
• After initial upload, the memory required for subsequent rsyncs dropped off significantly as expected
• Uploading ~1TB of data with the above took approx. 24h

Curious to see what others find from optimizing these further, but glad to have a working baseline in the mean time.

I'm also curious to see what people's monthly bills tally up to. Waiting for the first month's bill to come in but B2 seems more cost effective than Wasabi so far!

@Divemasterza figured out how to get this to work without making searx public (from https://forum.cloudron.io/topic/10618/how-to-connect-from-one-app-to-another-via-the-internal-network/6)

you can have cloudron oauth enabled but bypass it by calling its local hostname instead of public URL from openwebui. Tweaking @coniunctio 's query:

http://<searx app id>:<searx port>/search?q=<query>&language=auto&time_range=&safesearch=0&categories=social+media,map,it,general,science,news&format=json

example:

http://12345678-1234-1234-1234-abcdefabcdef:8888/search?q=<query>&language=auto&time_range=&safesearch=0&categories=social+media,map,it,general,science,news&format=json

find app id from cloudron dashboard, port from searxng filemanager->settings.yml under 'port:'

For future reference i managed to clear it up by using a slightly modified version of Girish's search command. It seems like the length limit applies to path length not just the filename length

find . -type f -print|awk '{print length($0), $0}' | sort -n

Went into the web interface and deleted / renamed files longer than 156 bytes and finally cleared trashbin/versioned files

(wipes all deleted / old versions of files - careful):

sudo -u www-data php -f /app/code/occ versions:cleanup
sudo -u www-data php -f /app/code/occ trashbin:cleanup

@robi @girish agreed it would be awesome if there was an automated way to handle this in the future

Use case-
Nextcloud updates very frequently, meaning that even if you only intend to keep 1-2 app backups at a time, you can end up with 5 or 6 due to cloudron's 3 week app update policy: https://docs.cloudron.io/backups/#retention-policy

If you have a 1tb nextcloud, this triples the storage costs / box space and adds up really quick. I think this is especially unnecessary when many people will also have a physical drive synced via the nextcloud client as a third backup.

You can delete these backups manually from your storage provider if you remember to, but it would be awesome to have an option to customize this, so people can enable automatic updates+backups of large apps in cloudron without driving too much cost.

@p44 i'm using rsync with hardlinks again, but via command line outside cloudron. i sync the entire backup folder but leave out --delete so it accumulates indefinitely on my home drive, something like

rsync -avH -e "ssh -p 202" --progress user@<backup-box-ip>:/backup/directory /local/directory

Honestly there's probably a better way... My drive will fill up eventually and all the daily/weekly backups aren't really necessary in cold storage. (Any suggestions?) I would also need to install cloudron locally if i ever want to access my files offline.

Have seen others mention restic / borg in the forum.

Wondering if there are any updates on whether mastodon will be in cloudron?

Would really like to see it

@joseph i think you might've mistaken my question for one about backups, but no worries you guys already had the answer here: https://docs.cloudron.io/apps/#data-directory

Hey All, circling back on this after a while, I don't think it was a permissions issue after all.

I suspect the way calibre desktop and calibre-web handle capitalization is slightly different and changes stored in metadata.db are case sensitive, so when I uploaded my library folder there were for whatever reason changes to the capitalization of author names that caused the issue.

Solved this by renaming folders and files in file manager to match what the error message was expecting, for example if the "Betty Smith" folder wasn't found, I renamed lowercase "betty smith" to "Betty Smith", after which I was able to normally use calibre-web. Had to do this for ~5-10% of books which is a pain but glad to have it working again. If anyone has found a better solution eager to hear it, scanning through the library for books with missing covers was a good indicator of where the broken links were.

@girish Ubuntu 16.04.4 LTS, ah I just spotted this article: https://docs.cloudron.io/guides/upgrade-ubuntu-20/ will try upgrading and see if the issue persists

Funkwhale looks awesome! Hope one day we can see it or something like it in cloudron and ditch Spotify

Hey all, is there a place this info can be found if your dashboard is down and you don't have access to the system info page / update section? Need to restore but unsure what version I was on around ~apr9th (7.6.___?)