Do not adjust your web browser. What follows is a retelling of an event that may or may not have taken place. This story is shared as a warning to others and for the general entertainment of anyone finding themselves with nothing better to do than to read random forum posts on a %current_day%. Enjoy.
It was raining cats and dogs (not literally) on a Sunday afternoon when I tried to run a backup on my Cloudron. My seat was warm and comfortable from having spent way too much time in it since I wrestled myself out of bed this morning. It provided soothing relief for what was to come.
At first, I suspected nothing wrong. Everything was as it seemed and no error messages lurking behind a page refresh to cause my stomach to feel upset.
It was backup restoration testing time and after readying myself with a fresh cup of PG Tips tea and a German cookie left over from the care parcel my mum sent for Christmas.
And then, my plans for a relaxing afternoon split between feeding the brood, killing NPCS in C&C Generals Zero Hour and waiting for backups to download went to hell!
The manually triggered backup was completed and I sunk into my chair in the same way a bowling ball does in a bean bag, not suspecting to be thrown down the aisle, meeting nine pins of doom head-on.
When I tried to restore my backup set, I was met with an error message I couldn't fathom. So I tried again. And again. But to no avail, the tar ball that supposedly rested safely in its volume was absent!
No backup was able to be restored. Frustrated, I topped up my tea and rummaged for more rapidly depleting supplies of Oma's finest Christmas cookies in an attempt to get to the bottom of this.
I remembered yesterday, my server ran out of disk space because I foolishly threw a ton of data at a volume that turned out to be way too small (imagine The IT Crowd episode where Jen is trying to cram her foot into a shoe that was way too small).
I tried removing files, emails anything that brought the space back to 47% available space, surely enough to run backups.
My remote drive didn't mount, so I switched to another using SSHFS rather than CIFS. Now nothing backs up anymore, my Nextcloud app lies broken - its logs claiming not to know who or what it was. The backups were completed without error pretending everything was OK when the reality was nothing close.
I was stumped. What should I do now? Only a few hours left in the day and dinner time was approaching fast.
I did what every sysadmin desperate enough would do: recover to a new instance with the last known backup set that was 'good'.
Good meaning the horrors and mistakes from the last 48 hours would be wiped out and soon to become nothing but a bad dream.
A dream too horrible to reimagine. I don't dare tell my children of the feeling of doom and dread, the shock that makes you stare into the middle distance for 9,000 yards, of a weekend not well spent.
But all this is behind me now and the children never need to know. My Cloudron runs happily on its new instance and the backups are purring.
As I slump back onto my sofa and my back meeting the comforting embrace of its pillows, I wonder what the next backup test restore day will bring and fall asleep.