Cloudron Forum

luckow

@imc67 one app instance (4y old)

+------+----------+
| hour | sessions |
+------+----------+
|    0 |        2 |
|    2 |        1 |
|    7 |        2 |
|    8 |        1 |
|    9 |        1 |
|   13 |        3 |
|   15 |        1 |
|   17 |        3 |
|   19 |        1 |
|   20 |        3 |
|   21 |        4 |
|   22 |        1 |
+------+----------+

different app instance (7y old)

+------+----------+
| hour | sessions |
+------+----------+
|    3 |        1 |
|    5 |        2 |
|   15 |        4 |
|   18 |        2 |
|   19 |        2 |
|   20 |        2 |
|   21 |        4 |
|   22 |        2 |
+------+----------+

health check is every 10 sec.

Mar 07 18:00:50 - - - [07/Mar/2026:17:00:50 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:00:50 172.18.0.1 - - [07/Mar/2026:17:00:50 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:00 - - - [07/Mar/2026:17:01:00 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:00 172.18.0.1 - - [07/Mar/2026:17:01:00 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:10 - - - [07/Mar/2026:17:01:10 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:10 172.18.0.1 - - [07/Mar/2026:17:01:10 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:20 - - - [07/Mar/2026:17:01:20 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:20 172.18.0.1 - - [07/Mar/2026:17:01:20 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:30 - - - [07/Mar/2026:17:01:30 +0000] "GET / HTTP/1.1" 302 - "-" "Mozilla (CloudronHealth)"
Mar 07 18:01:30 172.18.0.1 - - [07/Mar/2026:17:01:30 +0000] "GET / HTTP/1.1" 302 299 "-" "Mozilla (CloudronHealth)"

luckow

My two cents: as soon as #28 is correct, this should happen with every Cloudron instance that has Matomo (and OIDC enabled). I looked at one of my instances that met the criteria. One of the Matomo instances had about 300 sessions stored in MySQL. The oldest entry is from Feb 26.
So maybe #28 isn't correct, or it's something that only happens on this instance.

luckow

@timconsidine anything wrong on my side?

Installing App to diffy.example.org ...
Image: tcmbp132021/cloudron-dify:latest
✘ Installation failed: No such route
Details: {
  "status": "Not Found",
  "message": "No such route"
}

This it the output of curl -fsSL https://customappgateway.appx.uk/install.sh | bash -s -- --install-app "Dify AI" but same error from insie CCAI-P

girish

@timconsidine 9.1.3 is being rolled out . The upgrade path will be straight from 9.0.17 to 9.1.3

girish

@robi great find! will give it a try.

girish

@msbt yes, we have started the roll-out.

girish

@robi yeah, both play fine! That is why I was wishing this was a text file, so I could easily spot which is the correct one.

I actually have my pre-COVID data/backups burnt in DVDs But I don't have a DVD reader anymore.

girish

v8 is out but requires Cloudron 9.1

girish

We released backup integrity computation in 9.0 . In 9.1, we have added the verifier. At a high level, it stores sizes and hashes of the files in a file called .backupinfo alongside the backup itself. The .backupinfo is further checksummed and stored in the database so that it's signature can be verified when it's put to use.

The main intention was to catch bugs in the backup logic and also to theoretically detect bitrot. While testing, we already found some bugs in the rsync logic. There was a situation where the code will not delete non-existent files and the backup will end up with superfluous files.

Generally, my approach has been to blame the code whenever I see an integrity check fail. Today, I noticed that my own backups are failing integrity checks. Debugging further, I found that exactly one video file fails the check. The size of the file matches but the hash is different. I found this (built-in) tool called cmp:

# cmp VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
VID_20200712_143638936.mp4 /home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4 differ: byte 26595693, line 101836

OK, so it differs in byte 26595693 . I found that I can start checking other bytes from an offset using the -i arg.

# cmp -i 30000000 VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
<nothing>

OK, so it is fine from offset 30000000 to EOF (which was 84666735).

Bisecting slowly.... mostly because I thought I was going to find the unthinkable.. and I did!

# cmp -i 26595693 VID_20200712_143638936.mp4 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
<nothing>

Wow.. actual bitrot in offset 26595692 . Well , what's in that specific byte?

# xxd -b -l 1 -s 26595692 VID_20200712_143638936.mp4
0195d16c: 01010011                                               S
# xxd -b -l 1 -s 26595692 '/home/yellowtent/appsdata/19709657-2cf0-4d3f-8b79-429429d95b17/data/libraries/photos/USA - Jul 2020/VID_20200712_143638936.mp4'
0195d16c: 00010011                                               .

Wow, 1-bit flipped. I have never seen this in real life I wish this was a text file, because I don't know which is corrupt now - the backup or the original

This whole thing got me unreasonably excited, thanks for coming to my TED talk.

luckow

I followed https://docs.castopod.org/next/en/api/ but can't seem to connect to any endpoint. Maybe I just need a little kickstart.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

Partner

Posts

Member List