Cloudron Forum

andreasdueren

Can we please allow rsync for the lamp stack? That would speed up my CI/CD pipeline.

timconsidine

AI dev agent ? Which one ?
Just interested, doesn’t seem familiar.

girish

We released backup integrity computation in 9.0 . In 9.1, we have added the verifier. At a high level, it stores sizes and hashes of the files in a file called .backupinfo alongside the backup itself. The .backupinfo is further checksummed and stored in the database so that it's signature can be verified when it's put to use.

The main intention was to catch bugs in the backup logic and also to theoretically detect bitrot. While testing, we already found some bugs in the rsync logic. There was a situation where the code will not delete non-existent files and the backup will end up with superfluous files.

Generally, my approach has been to blame the code whenever I see an integrity check fail. Today, I noticed that my own backups are failing integrity checks. Debugging further, I found that exactly one video file fails the check. The size of the file matches but the hash is different. I found this (built-in) tool called cmp:

# cmp VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
VID_20200712_143638936.mp4 /home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4 differ: byte 26595693, line 101836

OK, so it differs in byte 26595693 . I found that I can start checking other bytes from an offset using the -i arg.

# cmp -i 30000000 VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
<nothing>

OK, so it is fine from offset 30000000 to EOF (which was 84666735).

Bisecting slowly.... mostly because I thought I was going to find the unthinkable.. and I did!

# cmp -i 26595693 VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
<nothing>

Wow.. actual bitrot in offset 26595692 . Well , what's in that specific byte?

# xxd -b -l 1 -s 26595692 VID_20200712_143638936.mp4
0195d16c: 01010011                                               S
# xxd -b -l 1 -s 26595692 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
0195d16c: 00010011                                               .

Wow, 1-bit flipped. I have never seen this in real life I wish this was a text file, because I don't know which is corrupt now - the backup or the original

This whole thing got me unreasonably excited, thanks for coming to my TED talk.

murgero

Wow! a lot of apps to choose from! I'll see what I can do this weekend with some of these

eyecreate

4.7.0 is released! I tested the rom patching and it seems to work now. I have nothing left to develop and get working now. Will do a polish pass on packaging, but hopefully this can either get into the cloudron app store or upcoming community app.

timconsidine

@LoudLemur said:

it's in a "works up to the last mile" state.

I've got a few of those, so understood

@LoudLemur said:

Would switching to base 5.0.0 and its native Node fix the indexing step too?

I've found that the answer to that type of question is only ever discovered by trying it !

I have a few other projects on/behind, but I might take a look. But please don't let that stop anyone else doing so.

timconsidine

@jdaviescoates Yes, I am planning to "convert" all my custom apps to Community Apps. But I'm generally cautious on new releases (no disrespect intended) so I am waiting for 9.1 to be stable before installing and cracking on (checking regularly).

But nothing to stop @murgero doing it before then..
I won't be offended in the slightest.
I seem to recall it wasn't a difficult one. So depending on perspective, an easy one to get back in the swing with, or not challenging enough to be interested in taking on.

timconsidine

@LoudLemur said:

What We Built
A Docker image based on cloudron/base:4.2.0

Why 4.2.0 ?
I use 5.0.0.
Supports more recent Node

If you're happy, share your repo and maybe someone can take a look.

timconsidine

@Joseph

Despite my few years on Cloudron and packaging some apps, I have never used the demo server ! Embarrassing. Must try harder.

So much more to know about Cloudron, so little time, such small brain.

timconsidine

@robi thank you
May not help others, but does help me.
Struggled a bit with SKILL for Nanobot, worked, then didn't, but now working correctly.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

AppDev

Posts

Member List