Full disk recovery process

[brerlapn]

I had a full disk last night - I was able to access the dashboard but the apps were all nonresponsive or causing an error. I managed to get into the app that had the most data and delete some larger files, and ran the Cleanup Backups. (Until then I was unable to launch the server console from DigitalOcean, though.) Three apps have resumed operating normally, but I'm still getting errors or failures to run for the app I had to manually delete files (Jirafeau) as well as from one other app (Shaarli). All services except 'unbound' are running green.

(Backups were running on the system disk, but retention period was set to 2 days so they shouldn't have filled up the disk. I've moved the backup location to an attached volume in the meantime.)

When I go to the backups screen from Cloudron settings, I see an error:

Unexpected response code or HTTP error when piping http://x.x.x.x:3000/databases .... postgresqldump: status 500

QUESTION: I'm running Shaarli but while Cloudrun indicates the app is running the page, it shows the error below when it's launched which sounds from googling as if the database is possibly corrupted. I have downloaded the Shaarli data directory to my local machine (the datastore.php file is over a MB so it seems like it's intact), but there's not much guidance about how to fix this - if I just uninstall Shaarli entirely and reinstall the app, can I copy the data directory over the newly installed data directory to get the app back to where it was yesterday morning?

The Shaarli page loads as:

An unexpected error happened, and the error template could not be displayed.

unserialize() expects parameter 1 to be string, bool given
/app/code/application/bookmark/BookmarkIO.php90
#0 /app/code/application/bookmark/BookmarkIO.php(90): unserialize()
#1 /app/code/application/bookmark/BookmarkFileService.php(71): Shaarli\Bookmark\BookmarkIO->read()
#2 /app/code/application/container/ContainerBuilder.php(97): Shaarli\Bookmark\BookmarkFileService->__construct()
#3 /app/code/vendor/pimple/pimple/src/Pimple/Container.php(118): Shaarli\Container\ContainerBuilder->Shaarli\Container\{closure}()
#4 /app/code/vendor/slim/slim/Slim/Container.php(109): Pimple\Container->offsetGet()
#5 /app/code/vendor/slim/slim/Slim/Container.php(164): Slim\Container->get()
#6 /app/code/application/container/ContainerBuilder.php(110): Slim\Container->__get()
#7 /app/code/vendor/pimple/pimple/src/Pimple/Container.php(118): Shaarli\Container\ContainerBuilder->Shaarli\Container\{closure}()
#8 /app/code/vendor/slim/slim/Slim/Container.php(109): Pimple\Container->offsetGet()
#9 /app/code/vendor/slim/slim/Slim/Container.php(164): Slim\Container->get()
#10 /app/code/application/front/controller/visitor/ErrorController.php(20): Slim\Container->__get()
#11 [internal function]: Shaarli\Front\Controller\Visitor\ErrorController->__invoke()
#12 /app/code/vendor/slim/slim/Slim/App.php(728): call_user_func_array()
#13 /app/code/vendor/slim/slim/Slim/App.php(396): Slim\App->handlePhpError()
#14 /app/code/vendor/slim/slim/Slim/App.php(297): Slim\App->process()
#15 /app/code/index.php(176): Slim\App->run()
#16 {main}

Jirafeau is the other app not running, and it has a persistent error even after executing the retry-configure task. If I have to uninstall and reinstall the app that's not a lot of re-work but I'm concerned that it became unrecoverable:

Docker Error: Error stopping container:(HTTP code 500) server error - cannot stop container: XXXXX: tried to kill container, but did not receive an exit event 

[nebulon]

Since you mention that unbound is not working, which is essential to the platform, this might be the root cause. Please try to follow the docs at https://docs.cloudron.io/troubleshooting/#unbound to get that up and running for a start.

[brerlapn]

@nebulon Thanks, the steps in the troubleshooting link you provided have gotten the unbound service running as green and active again, but Shaarli still launches to the error message I pasted above.

I'm also still seeing the "postgresqldump: status 500" error message when I go to the backup settings.

[nebulon]

@brerlapn can you maybe restore the shaarli app to the last backup if that is reasonable? Just to get a fresh start there, as from the error there is only information very app specific not much about Cloudron. Also you mentioned you had to delete some files, were those from shaarli?

[brerlapn]

@nebulon The files I deleted were only from Jirafeau, but Shaarli only shows the backup from last night (after the errors began) since I had to move the backup location to prevent the disk from filling up again. I believe previous backups were still in the vda1 partition - if I was able to copy them over to the current backup location would they show back up in the options for restoring the app? Are there any particular steps I'd need to take to do that?

Alternatively, if I were to reinstall the Shaarli application and copy the previous data folder into the installation, would that fix it or is the database error triggered by something in the data files?

My VPS backed up on 9/24, before the disk was full - If I just hit the "restore Droplet" option in the Digital Ocean console would that cause any issues for Cloudron?

[nebulon]

@brerlapn just copying over the backup files will not make them show up in the Cloudron UI. However you can use the app backup import feature to import from your previous shaarli backup from the disk as outlined in https://docs.cloudron.io/backups/#import-app-backup

Just copying data into an app manually usually does not work well with apps. I am no export on shaarli but I doubt it will just work.

Doing a full Droplet restore should also be fine, as long as you then quickly make sure the disk is not filling up again.

[brerlapn]

@nebulon Okay, thanks. Importing the backup wasn't completing successfully since I'd had to use the Clean Backups function to free up space.

I ended up just restoring the last backup image of the whole Droplet from DigitalOcean. That put things back as they were the night before the disk filled up so I only lost items from a single day.

I was relieved to find that once I'd gotten things back to a green status I was able to complete the setup process for an external backup in an S3 object storage account. I tried setting up Backblaze B2, DigitalOcean Spaces, and Wasabi on Sunday and Monday, but they always errored out with a complaint about the bucket not being available to the region the DO Droplet is located in. Monday I was able to get my DigitalOcean volume mount set as external backup, but obviously don't want to use block storage for backups. Once I'd restored the droplet and resolved the space issue, though, I was able to add B2 as external backup storage with no issues.

Thanks for the help!