I figured it out, my Cloudflare API token changed.

You are not wrong in principle, distributing workloads reduces blast radius, and that approach makes perfect sense in many scenarios. Our setup, however, is slightly different. We operate our own colocation facility (4K rack capacity) with direct, hands-on access to the hardware. For this particular server, there is a live mirror in place with continuous rsync-based synchronization, so the data is always effectively warm. Internally, the servers are connected over a dedicated 10 Gbps private network. In practical terms, this means restoring or promoting a mirror is not a lengthy operation. It is also worth noting that these instances are not customer facing. They support our internal workflows, not production services for external clients, so the operational impact of a short outage is limited by design. Even in the worst case, an instance going down does not trigger customer downtime or a flood of support calls. The main reason we run Cloudron on a single server is automation. We rely heavily on the Cloudron API for provisioning, backups, restores, and lifecycle management, and keeping everything under one Cloudron instance allows these processes to remain fully automated and consistent. But sure, it has some costs. Running multiple Cloudron instances on separate servers would significantly increase operational complexity and require manual coordination, which defeats the purpose of our setup. Given our mirrored data, high-speed internal networking, and automated recovery, a single Cloudron control panel is a deliberate and acceptable trade-off for our use case.

@joseph said in Can't see any file in the file manager & SFTP broken: have you tried to restart SFTP in Services -> SFTP already? I hadn't, and I noticed that SFTP service was in recovery mode for some reason (I'm going to try to figure out why, but it's most probably a person issue and not a technical one). Restarting it worked. I should have checked there before asking.

The dialog is also fixed now with https://git.cloudron.io/platform/box/-/commit/612002ec33f54d23ab4e4c38e0e5e0d3853e9db2

It works now

Also note, I am waiting for 9.0.14 to be stable release before I can update and then download a backup. Currently it's not possible to download a backup in 9.0.13 from Cloudflare R2.

Is the question whether this is allowed? If so, yes. As @jdaviescoates said, the Cloudron version needs to match. The OS version can be different. As a precautionary measure take a VPS snapshot of your old 22.04 install before migrating.

@girish Thank you very much for the info. I didn’t manage to reply earlier, but I saved the backup configuration again and the morning backup completed successfully. I’ll also try upgrading to 9.0.15.

@dylightful If we can have access to your setup, can you write to me at support@cloudron.io ? Would love to debug and figure out why it's failing . Is this reproducible easily - meaning if you take a backup and restore, it fails?

@nebulon That's great. Apart from that, it seems to be running smoothly; it still takes time to propagate the usage stats, but that doesn't really affect us.

@archos said in Old system backups are not listed: and restoring an older app backup fails with the fsmetadata.json ENOENT error. Does this file exist in the storage?

There is no look-into-it. There is only do-it.

Hello @imc67 Now we can start analysing. Edit the file /home/yellowtent/platformdata/mysql/custom.cnf and add the following lines: [mysqld] general_log = 1 slow_query_log = 1 Restart the MySQL service in the Cloudron Dashboard. The log files are stored at /home/yellowtent/platformdata/mysql/mysql.log and /home/yellowtent/platformdata/mysql/mysql-slow.log. Let it run for a day or more. Then you can download the log files and see what queries run very often causing disk I/O.

@joseph ~~Now I have reseted the router and I still have the same problem. I guess it's my ISPs Cgnat that is the problem. It's strange. It has been working for almost a year with the same ISP and the Cgnat but suddenly it stopped to work. I guess it's the Cgnat that make it impossible to ping the public ip too. My apps with Cloudflare tunnels is working but the tunnel I created to the Cloudron dashboard gets bad gateway and host error from Cloudflare. What am I missing?~~ Now it's working. It was simple when I found it. I had forgot to activate No TLS Verify for the tunnel. Now it's working. Thank you for your help.

Ok, I reached out to Vultr Wednesday about it and they instructed me to contact Ya'll directly. Did a fresh install and it installed 8.0. Glad to see it's updated.

@philkunz said in Backups still not verified or what?: But that points to how important the e2e read back step is. absolutely, that is why we are adding the validation in the first place. Having checksums of what we expect is the first step, next then will be the task (can be quite heavy task, so this won't be automatic probably) to read back and recalculate the hashes.

I investigated this on support@ . The issue was that the server had an incredibly slow disk and the iops was really bad (like 30MB/s) . The VPS provider is fixing their raid configuration .

@james Your support is always unbelievably great - thank you so much!

The schema is documented in this file - https://git.cloudron.io/platform/box/-/blob/master/migrations/schema.sql?ref_type=heads . completed BOOLEAN DEFAULT false The id has to be unique in oidcClients table. Can you check why you have a duplicate? Maybe you can delete the dup entry and run the migration again.