Any ideas?
OrezBodajoy
Posts
-
Access monitoring (login events, suspicious activity detection) -
Access monitoring (login events, suspicious activity detection)I want to monitor access to my cloudron instance: login events and things like that.
I also want to sometimes clean up all the currently valid credentials and check that there are no unexpected logons after that.I know that the event log has some lines in the ui related to logons but that's not practical enough for this use case.
Any suggestions?
-
Installing Firefly III Data Importer on Cloudron – Vite Manifest and Secret IssuesI got this working by adapting the instructions in the doc by @JLX89
It's pretty much the same thing, except:
- set PhP version 8.4
- do NOT run
composer create-project firefly-iii/data-importer - open the cloudron terminal and download the release there by typing
wget https://github.com/firefly-iii/data-importer/releases/download/v1.9.0/DataImporter-v1.9.0.tar.gz - extract that to
/app/data/data-importerby doing
tar -xvf DataImporter-v1.9.0.tar.gz -C /app/data/data-importer - continue the setup:
sudo chown -R www-data:www-data /app/data/data-importer sudo chmod -R 775 /app/data/data-importer/storage ...The changes were made following this https://docs.firefly-iii.org/how-to/data-importer/installation/self-managed/
-
Is there a way to skip dns propagation check after app install or change detected IP address?It worked! Thanks.
-
Is there a way to skip dns propagation check after app install or change detected IP address?I am running cloudron at home without using my home public address. Instead, I am reverse-proxying to a VPS with a static IP address, and my domain's IP address resolves to that VPS IP address.
By default, the detected IP address on my cloudron installation is my home IP address. After I install an app, I get a phase where cloudron tries to check that the new app's sub-domain points to the right IP address.
I am using manual DNS with a catch-all sub-domain by the way.
Cloudron tries to match the records and fails because my domain's IP address does not match the detected IP address on my cloudron.I know I can setup a vpn for my cloudron so it thinks the detected IP address is the one of my VPS, but that is slower and a bit overkill...
Is there a way to skip that check or just set the IP address detected by cloudron?
Thanks.
-
queryNs ESERVFAIL for new appsThanks for your input @joseph, I have found the issue!
It turns out that I am running cloudron in a qemu vm, and qemu has a bug in the default config where it will not resolve dns correctly if the first name server configured on the host fails.
In that case, it will not try the second nameserver configured on the host. DNS resolution will work on the host, but not on the guest.
Here is the link: https://unix.stackexchange.com/a/614603 -
queryNs ESERVFAIL for new appsSorry for the delay, I can now check this issue.
Please note that the issue seems to be slightly different, I have moved the server's disk image and it seems that now unbound is working but not systemd-resolve.
Here are some command results.
redacted@cloudron:~$ ping 9.9.9.9 PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data. 64 bytes from 9.9.9.9: icmp_seq=1 ttl=255 time=16.4 ms 64 bytes from 9.9.9.9: icmp_seq=2 ttl=255 time=15.6 ms 64 bytes from 9.9.9.9: icmp_seq=3 ttl=255 time=22.8 ms 64 bytes from 9.9.9.9: icmp_seq=4 ttl=255 time=14.2 ms 64 bytes from 9.9.9.9: icmp_seq=5 ttl=255 time=15.3 ms ^C --- 9.9.9.9 ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4006ms rtt min/avg/max/mdev = 14.202/16.835/22.778/3.052 msredacted@cloudron:~$ host -t NS myredacteddomain.com ;; communications error to 127.0.0.53#53: timed out ;; communications error to 127.0.0.53#53: timed out ;; no servers could be reached redacted@cloudron:~$ host -t NS myredacteddomain.com 127.0.0.150 ;; communications error to 127.0.0.150#53: timed out Using domain server: Name: 127.0.0.150 Address: 127.0.0.150#53 Aliases: myredacteddomain.com name server 2-can.njalla.in. myredacteddomain.com name server 1-you.njalla.no. myredacteddomain.com name server 3-get.njalla.fo. redacted@cloudron:~$ systemctl status unbound.service WARNING: terminal is not fully functional Press RETURN to continue ● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2025-10-13 05:03:53 UTC; 10h ago Process: 752 ExecStartPre=/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key (code=exited, status=0/SUCCESS) Main PID: 1002 (unbound) Tasks: 1 (limit: 17831) Memory: 11.1M CPU: 599ms CGroup: /system.slice/unbound.service └─1002 /usr/sbin/unbound -d Oct 13 15:43:10 cloudron unbound[1002]: [1002:0] info: generate keytag query _ta-4f66-9728. NULL IN Notice: journal has been rotated since unit was started, output may be incomplete. redacted@cloudron:~$ lsof -i :53redacted@cloudron:~$ lsof -i :53 redacted@cloudron:~$ cat /etc/unbound/unbound.conf # Unbound configuration file for Debian. # # See the unbound.conf(5) man page. # # See /usr/share/doc/unbound/examples/unbound.conf for a commented # reference config file. # # The following line includes additional configuration files from the # /etc/unbound/unbound.conf.d directory. include-toplevel: "/etc/unbound/unbound.conf.d/*.conf" redacted@cloudron:~$ ll /etc/unbound//unbound.conf.d/ total 20 drwxr-xr-x 2 root root 4096 Sep 19 18:28 ./ drwxr-xr-x 3 root root 4096 Jul 23 06:24 ../ -rw-r--r-- 1 root root 692 Sep 15 17:33 cloudron-network.conf -rw-r--r-- 1 root root 142 Sep 10 19:26 forward-everything.conf.moved -rw-r--r-- 1 root root 190 Sep 7 2022 root-auto-trust-anchor-file.conf redacted@cloudron:~$ cat /etc/unbound/unbound.conf.d/cloudron-network.conf # Unbound is used primarily for RBL queries (host 2.0.0.127.zen.spamhaus.org) # We cannot use dnsmasq because it is not a recursive resolver and defaults to the value in the interfaces file (which is Google DNS!) server: port: 53 interface: 127.0.0.150 interface: 172.18.0.1 ip-freebind: yes do-ip6: yes access-control: 127.0.0.1 allow access-control: 172.18.0.1/16 allow cache-max-negative-ttl: 30 cache-max-ttl: 300 # enable below for logging to journalctl -u unbound # verbosity: 5 # LOG-QUERIES: yes # https://github.com/NLnetLabs/unbound/issues/806 remote-control: control-enable: no~$ systemctl status systemd-resolvectl WARNING: terminal is not fully functional Press RETURN to continue Unit systemd-resolvectl.service could not be found. :~$ systemctl status systemd-resolved WARNING: terminal is not fully functional Press RETURN to continue ● systemd-resolved.service - Network Name Resolution Loaded: loaded (/lib/systemd/system/systemd-resolved.service; enabled; vendor preset: enabled) Active: active (running) since Mon 2025-10-13 16:20:15 UTC; 4min 55s ago Docs: man:systemd-resolved.service(8) man:org.freedesktop.resolve1(5) https://www.freedesktop.org/wiki/Software/systemd/writing-network-configuration-managers https://www.freedesktop.org/wiki/Software/systemd/writing-resolver-clients Main PID: 302750 (systemd-resolve) Status: "Processing requests..." Tasks: 1 (limit: 17831) Memory: 8.8M CPU: 1.013s CGroup: /system.slice/systemd-resolved.service └─302750 /lib/systemd/systemd-resolved Oct 13 16:23:29 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3. Oct 13 16:23:34 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3. Oct 13 16:23:45 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3. Oct 13 16:23:50 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3. Oct 13 16:24:10 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3. Oct 13 16:24:15 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3. Oct 13 16:24:25 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3. Oct 13 16:24:30 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3. Oct 13 16:24:50 cloudron systemd-resolved[302750]: Using degraded feature set UDP instead of TCP for DNS server 10.0.2.3. Oct 13 16:24:55 cloudron systemd-resolved[302750]: Using degraded feature set TCP instead of UDP for DNS server 10.0.2.3.~$ resolvectl status WARNING: terminal is not fully functional Press RETURN to continue Global Protocols: -LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported resolv.conf mode: stub Link 2 (enp0s9) Current Scopes: DNS Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported Current DNS Server: 10.0.2.3 DNS Servers: 10.0.2.3Note, on this page in the documentation I see the command
systemctl status systemd-resolved
It should besystemctl status systemd-resolvectlright? -
queryNs ESERVFAIL for new appssystemctl status unbound.servicesays active and running, andlsof -i :53has no output at allI will post the full output of the first command
● unbound.service - Unbound DNS Resolver Loaded: loaded (/etc/systemd/system/unbound.service; enabled; vendor preset: enabled) Active: active (running) since Wed 2025-09-24 10:56:03 UTC; 6h ago Process: 755 ExecStartPre=/usr/sbin/unbound-anchor -a /var/lib/unbound/root.key (code=exited, status=0/SUCCESS) Main PID: 6775 (unbound) Tasks: 1 (limit: 12995) Memory: 9.6M CPU: 365ms CGroup: /system.slice/unbound.service └─6775 /usr/sbin/unbound -d Notice: journal has been rotated since unit was started, output may be incomplete.Here is the output of
netstat -ntlp | grep 53tcp 0 0 172.18.0.1:53 0.0.0.0:* LISTEN 6775/unbound tcp 0 0 127.0.0.53:53 0.0.0.0:* LISTEN 715/systemd-resolve tcp 0 0 10.0.2.15:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 10.0.2.15:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 10.0.2.15:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 10.0.2.15:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 127.0.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 127.0.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 127.0.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 127.0.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 127.0.0.150:53 0.0.0.0:* LISTEN 6775/unbound tcp 0 0 172.18.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.18.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.18.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.18.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.17.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.17.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.17.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp 0 0 172.17.0.1:5349 0.0.0.0:* LISTEN 1444/turnserver tcp6 0 0 fec0::5054:ff:fe12:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fec0::5054:ff:fe12:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fec0::5054:ff:fe12:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fec0::5054:ff:fe12:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fd00:c107:d509::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fd00:c107:d509::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fd00:c107:d509::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 fd00:c107:d509::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 ::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 ::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 ::1:5349 :::* LISTEN 1444/turnserver tcp6 0 0 ::1:5349 :::* LISTEN 1444/turnserver -
queryNs ESERVFAIL for new appsI got this
Vendor: QEMU Product: Standard PC (Q35 + ICH9, 2009) Linux: 5.15.0-153-generic Ubuntu: jammy 22.04 Processor: AMD Ryzen 5 PRO 2400G with Radeon Vega Graphics x 4 RAM: 11205884KB Disk: /dev/mapper/ubuntu--vg-ubuntu--lv 23G [OK] node version is correct [OK] IPv6 is enabled in kernel. No public IPv6 address [OK] docker is running [OK] docker version is correct [OK] MySQL is running [OK] nginx is running [OK] dashboard cert is valid [OK] dashboard is reachable via loopback [OK] box v8.3.2 is running [OK] netplan is good [OK] DNS is resolving via systemd-resolved [OK] Dashboard is reachable via domain name [WARN] Domain myredacteddomain.com expiry check skipped because whois does not have this information [FAIL] Unbound is not resolving, maybe try forwarding all DNS requests. You can do this by running 'cloudron-support --unbound-use-external-dns' option ;; communications error to 127.0.0.150#53: timed out ;; communications error to 127.0.0.150#53: timed out ;; no servers could be reachedand when I try the command suggested to forward everything, I get
/usr/bin/cloudron-support: line 803: unbound_use_external_dns: command not found -
queryNs ESERVFAIL for new appsI am not able to deploy new apps, I am on Ubuntu 22.
host -t NS mydomain.comworks fine.
host -t NS mydomain.com 127.0.0.150does not work (timeout or not reachable IIRC).The unbound service says it's active and running, restarting it does not fix the issue.
unbound-control says control is not enabled, and the service is stopped.I also tried removing the forward everything unbound config and restarting.
Do you know a way to test if unbound is working correctly?
-
How to change the default admin password?Maybe I am missing something obvious, but I don't see how to change the default admin password.
This link says to use a postgre user but that user does not exist in the cloudron web terminal.
-
Backups keep failingThanks @jdaviescoates ! I will try that!
@nebulon I just checked the system logs and there are a bunch of out of memory error logs:
Out of memory: Killed process XXXXX (ruby2.3 - and sometimea node) total-vm: 1704896kB, anon-rss: 891324kB, file-rss: 0kB, shmem-rss: 0kB, UID: XXXX pgtables: 2072kB, ooom_score_adj: 0I will check to see if there are still some backup memory settings I need to increase.
-
Backups keep failing@nebulon is there a way to update the app without doing the backup first?
I am now running a 6+ month old version without backups and this is really getting annoying now.Note: switching to AWS S3 and changing the backup config has improved the situation but I still get an error during backup after a few hours.
Here is my log for reference. I only have a "Task 4059 crashed with code 1", I don't know what caused it.
https://upload.disroot.org/r/B82DZPjw#3JC3Iw05uQQVfACMKuVNPBtDjYha2BfXiKaSC82w3+E=Is there no way to know what's causing the issue?
-
Backups keep failing@mdc773 you mean https://www.idrive.com/s3-storage-e2/ ? I will give it a try, thanks for the suggestion.
@nebulon yes. I am using a vpn, maybe it's linked to that.
-
Backups keep failingQuick update, sorry for the delay.
After migrating from backblaze to aws s3 it seems I no longer have the EPIPE errors.Here is what I have on the latest failed backup logs:
Mar 04 01:53:23 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"} Mar 04 01:53:29 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/2f/1d/2f1d57dd-f397-4715-a214-96838b38d205.mp4: 46M@0MBps (photos.mysuper.website)"} Mar 04 01:53:37 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/30/f8/30f835f2-08c2-4366-9915-fd4a92064959.mp4: 46M@0MBps (photos.mysuper.website)"} Mar 04 01:53:43 box:tasks update 3297: {"percent":22.05263157894737,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/32/20/3220fbd9-40e6-4d16-91ef-fce5bdc0913d.mp4: 46M@0MBps (photos.mysuper.website)"} [no timestamp] /home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70 [no timestamp] new Error('Socket timed out without establishing a connection'), [no timestamp] ^ [no timestamp] [no timestamp] Error [TimeoutError]: Socket timed out without establishing a connection [no timestamp] at Timeout.connectTimeout [as _onTimeout] (/home/yellowtent/box/node_modules/aws-sdk/lib/http/node.js:70:15) [no timestamp] at listOnTimeout (node:internal/timers:581:17) [no timestamp] at process.processTimers (node:internal/timers:519:7) { [no timestamp] code: 'TimeoutError', [no timestamp] time: 2025-03-04T00:53:48.827Z, [no timestamp] region: 'eu-central-1', [no timestamp] hostname: 'impressive-dangly-smut.s3.eu-central-1.amazonaws.com', [no timestamp] retryable: true, [no timestamp] statusCode: 200 [no timestamp] } [no timestamp] [no timestamp] Node.js v20.18.0 Mar 04 01:54:07 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28) Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12) Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Mar 04 01:54:07 reason: 'Shell Error', Mar 04 01:54:07 details: {}, Mar 04 01:54:07 code: 1, Mar 04 01:54:07 signal: null Mar 04 01:54:07 } Mar 04 01:54:07 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Mar 04 01:54:07 at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:137:19) Mar 04 01:54:07 at ChildProcess.emit (node:events:519:28) Mar 04 01:54:07 at ChildProcess.emit (node:domain:488:12) Mar 04 01:54:07 at ChildProcess._handle.onexit (node:internal/child_process:294:12) { Mar 04 01:54:07 reason: 'Shell Error', Mar 04 01:54:07 details: {}, Mar 04 01:54:07 code: 1, Mar 04 01:54:07 signal: null Mar 04 01:54:07 } Mar 04 01:54:07 box:backuptask fullBackup: app photos.mysuper.website backup finished. Took 4302.722 seconds Mar 04 01:54:07 box:locks write: current locks: {"backup_task":null} Mar 04 01:54:07 box:locks release: app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 Mar 04 01:54:07 box:taskworker Task took 6845.852 seconds Mar 04 01:54:07 box:tasks setCompleted - 3297: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Mar 04 01:54:07 box:tasks update 3297: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Mar 04 01:54:07 BoxError: Backuptask crashed Mar 04 01:54:07 at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) Mar 04 01:54:07 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Mar 04 01:54:07 at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) Mar 04 01:54:07 at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) -
Backups keep failingI am trying now with AWS.
I didn't get an epipe error but the "Task 3061 timed out" after 1h exactly.
My photo album is quite big and I don't expect it to be uploaded within an hour. Is there a way to increase this timeout limit for the backup task? I don't see anything about that in the advanced section of the backup configuration.Thanks
-
Backups keep failingPlatform versionv8.2.3 (Ubuntu 22.04.4 LTS)
I will try updating to 8.2.4Would it work better if I use another backup method than backblaze?
-
Backups keep failingI have changed a bit the backup parameters in the UI:
- memory limit to 9 GiB
- upload part size 64 MiB
- concurrency items 10
I now have this error:
Feb 04 13:48:29 box:storage/s3 Upload progress: {"loaded":62003204,"total":62003204,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ"} Feb 04 13:48:32 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4","reason":"new","position":425} Feb 04 13:48:32 box:storage/s3 Upload finished. {"ETag":"\"96b07711e9e9cd6313ae1d3bf2eab89d\"","VersionId":"4_z30c7662aa70ee25089da0419_f112b8445e8479200_d20250204_m124750_c003_v0312024_t0049_u01738673270973","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc%2B63z%2BGW%2BIIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/0KPhbUuBjncr7PGqgZwzB4fmP2RhEubtj8cyGzIjOgc/P1wybT781lXr7wdAaxpbhPW9khj0bqxBeCNwK00FB18/xY3HW0Fc9c3GoFxrS7ipncSqa80AEMVc+63z+GW+IIMmG24hCz8ddQgvpP45fpFhQ9dRfXTcMloEea9KhYFzYQ","Bucket":"cloudron-backup-january-february"} Feb 04 13:48:32 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 position 425 try 1 Feb 04 13:48:32 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f6/e3/f6e37991-d39d-4bab-9317-afca881eda91.mp4 (photos.mysuper.website)"} Feb 04 13:48:34 box:storage/s3 Upload progress: {"loaded":12934596,"total":12934596,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw"} Feb 04 13:48:37 box:storage/s3 Upload progress: {"loaded":24622868,"total":24622868,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA"} Feb 04 13:48:38 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4","reason":"new","position":426} Feb 04 13:48:38 box:storage/s3 Upload finished. {"ETag":"\"a63eb3273bd8c6af9c8be8cdb8b3411c\"","VersionId":"4_z30c7662aa70ee25089da0419_f111b63a9bf79370d_d20250204_m124018_c003_v0312024_t0006_u01738672818986","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-%2Bbm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF%2BWvX4eM5wpaPa04Cp9mm0g9jyU%2BWHPQ","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/e4LlBfnRs2P83jBNmVfv-h8S-snRVHnPmaS4dtZW2vY/ePMwmyFhRxqFNN7QBJzlCWOFsK-Qvsw42NbUryxdEiU/7NKck9Wkh-+bm1O0jmO6SyT3eqB1rwGEE5nHJje5Vc2nihu8hYC7UyF+WvX4eM5wpaPa04Cp9mm0g9jyU+WHPQ","Bucket":"cloudron-backup-january-february"} Feb 04 13:48:38 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 position 426 try 1 Feb 04 13:48:38 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/14/f7145cfc-2306-45f1-92f7-5a19add3aa9c.mp4 (photos.mysuper.website)"} Feb 04 13:48:43 box:storage/s3 Upload finished. {"ETag":"\"e8f727fbe3017a736863acd3e005920f\"","VersionId":"4_z30c7662aa70ee25089da0419_f102a9d6210f3b68d_d20250204_m124644_c003_v0312028_t0032_u01738673204217","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/DBIzrCxVqVk3kzOgs2UfGVfVwvfHWqLVm-Fvk6CA-e4/Xz7ApaTu7-QxV5ai4eLprhxdYTI9yssL86kbCdFi9FM/IV6dab6-MyVWBPeF8FJ8ItaEjR8kqXcTxI5pmszMhqFzG2uEA053FqFdR34oPN77RWkwkhadsc2SqtEYbSSlvw","Bucket":"cloudron-backup-january-february"} Feb 04 13:48:43 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4 position 427 try 1 Feb 04 13:48:43 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/70/f770093e-d633-4553-8e18-76928a64abca.mp4","reason":"new","position":427} Feb 04 13:48:45 box:storage/s3 Upload finished. {"ETag":"\"fef4d9e41a7b0487ef85cde77de337c7\"","VersionId":"4_z30c7662aa70ee25089da0419_f105b54dc655def0e_d20250204_m124514_c003_v0312025_t0014_u01738673114140","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE%2BWncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3%2BwvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/8PCmzF3EVYPE+WncvS7A7V4qP9u3qm3s6NSg0bNwB4s/XaenUbcaa72K2xvF3+wvbOb8sS2ACmIApKq47zKPb9I/RKfaJr9nuoGi6m7KLX-bABpNePzu2pgw0qXGosmKk1kzAzwZ9SKPxuVP8KnXvDpPHZRdrwWeSAQoKVE4PX1NUA","Bucket":"cloudron-backup-january-february"} Feb 04 13:48:45 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4","reason":"new","position":428} Feb 04 13:48:45 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 position 428 try 1 Feb 04 13:48:45 box:tasks update 3016: {"percent":30,"message":"Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f7/d8/f7d8868b-8af8-489d-a6e8-3ba76e06f810.mp4 (photos.mysuper.website)"} Feb 04 13:48:48 box:storage/s3 Upload progress: {"loaded":83903444,"total":83903444,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg"} Feb 04 13:48:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:48:56 zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83%2BN073h%2BYeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","Bucket":"cloudron-backup-january-february","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/7TW2OB1TBrQ57XB3jzn8HgA7Nke74zrlTG7lc0704xk/w1cW6moKv9lYkZgZDxUkRN4Zxl8myZ-CEvE3Ye-HQhg/gb9sGl1g7pgYmuGbcL83+N073h+YeIRnZK2s9dbPK5wxKEhNFw59MZiZqfU2f9QieTqgP-tXt9Njcir3yGQYEg","ETag":"\"de497871d785fd1c59918d801ccfa714-2\""} Feb 04 13:48:56 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4 position 429 try 1 Feb 04 13:48:56 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f8/f3/f8f30130-24cd-47e8-872a-c804bd9ca9da.mp4","reason":"new","position":429} Feb 04 13:49:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:49:11 box:storage/s3 Upload progress: {"loaded":51287460,"total":118396324,"part":2,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/JXzM88Uqx7ZsSnmpRN00J7pjTkogYPeJj9sRGLylY-Y/lCpIzgRkjCJJlzwTqL5scjzLiKZZcSHPgHnseJC+9VU/xzreWsN82y+-8efk7r1FwykGI9qCIdoC1+WVWtIjKuolkKXYiFi4nMAPlEa4HbLqzGkhp4PA7nDfRQZSvpQgNA"} Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.044 * 10 changes in 300 seconds. Saving... Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.046 * Background saving started by pid 248 Feb 04 13:49:12 13:M 04 Feb 2025 12:49:12.146 * Background saving terminated with success Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.129 * DB saved on disk Feb 04 13:49:12 248:C 04 Feb 2025 12:49:12.130 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 13:49:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:49:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:49:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:49:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:49:56 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:06 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:12 box:storage/s3 Upload progress: {"loaded":79710836,"total":146819700,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/LQs0pg3EACKpxLfIoyhm3-pwf2CjUC0Qmr45cpZ0XtY/zRON9tC4HPaxKx9vNEkZkXGSyTtIBEVsHwJTKqt6EAI/zSGMZ1+pEr5om7CkzelvYADDA4IPqQHHBv18fQYz-AzjLcjdhD8N9ce0c7KaGaYbYwyPXwq5IcGX07SjbJQWzw"} Feb 04 13:50:16 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:26 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:36 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:36 box:storage/s3 Upload progress: {"loaded":21172484,"total":21172484,"part":1,"key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q"} Feb 04 13:50:39 box:backupformat/rsync Adding data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4 position 430 try 1 Feb 04 13:50:39 box:backupformat/rsync sync: processing task: {"operation":"add","path":"data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f9/84/f984597c-729e-406f-ad68-73275081a91b.mp4","reason":"new","position":430} Feb 04 13:50:39 box:storage/s3 Upload finished. {"ETag":"\"e8db66fe56ced58e8bccbf20014a6b92\"","VersionId":"4_z30c7662aa70ee25089da0419_f1064034b6c53492b_d20250204_m124857_c003_v0312028_t0047_u01738673337237","Location":"https://cloudron-backup-january-february.s3.eu-central-003.backblazeb2.com/snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k%2B5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg%2BwW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ%2BUS7GhiAPkvxud%2BGhx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0%2BgncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy%2BIEnSGqo17KBKZjlQcIp7Q","key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Key":"snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076/By8169k+5t6ApU5J9RRxIHnhEdUt9ZLcwMdOzzg4zaY/oEtU3dvIl7fyle9hukEV1y6EC9fxUgsZDIZvg+wW5Qg/axJfbCsBMvfcT8e8Uy-FZW8CPjKDUUBAaFaBe4bx9jg/Xe4iKBFLF9isOK0zcwdxWiDMGoCzdnvmJ+US7GhiAPkvxud+Ghx0RTG3jfbRrboc-texm4iEKjFt9iuZo2WA3Q/SOlrDfP9m6u2e4I5p44PIDh8HSg0IPGy2LPaDZ72PEo/l0+gncIhXvicAKKVNFY-njrXTET921WkntYTifcK7yE/d0AO9QuFyiYOpalIzyM0LA9WAyPtS9faM719jCVO5oR81AAyJiG9gzNp4pwWaUjy+IEnSGqo17KBKZjlQcIp7Q","Bucket":"cloudron-backup-january-february"} Feb 04 13:50:45 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null Feb 04 13:50:46 box:tasks update 3016: {"percent":30,"message":"Uploading /home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076/data/upload/encoded-video/177472c9-60f1-4746-a2c2-fd947abb8cca/f5/56/f556116d-2ae2-4a13-bd53-bf315134c7ae.mp4: 256M@0MBps (photos.mysuper.website)"} Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null Feb 04 13:50:46 box:backupformat/rsync Attempt 1 failed. Will retry: Upload error: code: 408 message: null Feb 04 13:50:46 box:shell backuptask: /usr/bin/sudo -S -E --close-from=4 /home/yellowtent/box/src/scripts/backupupload.js snapshot/app_14dd2bbe-1d61-4f3d-91d3-4af16e366076 rsync {"localRoot":"/home/yellowtent/appsdata/14dd2bbe-1d61-4f3d-91d3-4af16e366076","layout":[]} errored BoxError: backuptask exited with code 1 signal null Feb 04 13:50:46 box:backuptask runBackupUpload: backuptask crashed BoxError: backuptask exited with code 1 signal null Feb 04 13:50:46 box:apptask run: app error for state pending_backup: BoxError: Backuptask crashed at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15) at process.processTicksAndRejections (node:internal/process/task_queues:95:5) at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5) at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5) at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12) at async backup (/home/yellowtent/box/src/apptask.js:400:22) { reason: 'Internal Error', details: {} } Feb 04 13:50:46 box:taskworker Task took 12148.089 seconds Feb 04 13:50:46 box:tasks setCompleted - 3016: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Feb 04 13:50:46 box:tasks update 3016: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at runBackupUpload (/home/yellowtent/box/src/backuptask.js:170:15)\n at process.processTicksAndRejections (node:internal/process/task_queues:95:5)\n at async uploadAppSnapshot (/home/yellowtent/box/src/backuptask.js:369:5)\n at async backupAppWithTag (/home/yellowtent/box/src/backuptask.js:391:5)\n at async Object.backupApp (/home/yellowtent/box/src/backuptask.js:329:12)\n at async backup (/home/yellowtent/box/src/apptask.js:400:22)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}} Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.044 * 10 changes in 300 seconds. Saving... Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.046 * Background saving started by pid 249 Feb 04 13:54:13 13:M 04 Feb 2025 12:54:13.147 * Background saving terminated with success Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.137 * DB saved on disk Feb 04 13:54:13 249:C 04 Feb 2025 12:54:13.139 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.020 * 10 changes in 300 seconds. Saving... Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.022 * Background saving started by pid 250 Feb 04 13:59:14 13:M 04 Feb 2025 12:59:14.223 * Background saving terminated with success Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.124 * DB saved on disk Feb 04 13:59:14 250:C 04 Feb 2025 12:59:14.125 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.094 * 10 changes in 300 seconds. Saving... Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.097 * Background saving started by pid 251 Feb 04 14:04:15 13:M 04 Feb 2025 13:04:15.198 * Background saving terminated with success Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.188 * DB saved on disk Feb 04 14:04:15 251:C 04 Feb 2025 13:04:15.189 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.081 * 10 changes in 300 seconds. Saving... Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.082 * Background saving started by pid 252 Feb 04 14:09:16 13:M 04 Feb 2025 13:09:16.285 * Background saving terminated with success Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.184 * DB saved on disk Feb 04 14:09:16 252:C 04 Feb 2025 13:09:16.185 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.059 * 10 changes in 300 seconds. Saving... Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.061 * Background saving started by pid 253 Feb 04 14:14:17 13:M 04 Feb 2025 13:14:17.262 * Background saving terminated with success Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.160 * DB saved on disk Feb 04 14:14:17 253:C 04 Feb 2025 13:14:17.162 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.096 * 10 changes in 300 seconds. Saving... Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.098 * Background saving started by pid 254 Feb 04 14:19:18 13:M 04 Feb 2025 13:19:18.198 * Background saving terminated with success Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.175 * DB saved on disk Feb 04 14:19:18 254:C 04 Feb 2025 13:19:18.177 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.048 * 10 changes in 300 seconds. Saving... Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.049 * Background saving started by pid 255 Feb 04 14:24:19 13:M 04 Feb 2025 13:24:19.149 * Background saving terminated with success Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.142 * DB saved on disk Feb 04 14:24:19 255:C 04 Feb 2025 13:24:19.144 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.087 * 10 changes in 300 seconds. Saving... Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.089 * Background saving started by pid 256 Feb 04 14:29:20 13:M 04 Feb 2025 13:29:20.190 * Background saving terminated with success Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.163 * DB saved on disk Feb 04 14:29:20 256:C 04 Feb 2025 13:29:20.165 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.052 * 10 changes in 300 seconds. Saving... Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.053 * Background saving started by pid 257 Feb 04 14:34:21 13:M 04 Feb 2025 13:34:21.154 * Background saving terminated with success Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.134 * DB saved on disk Feb 04 14:34:21 257:C 04 Feb 2025 13:34:21.136 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB Feb 04 14:34:21 408: null Feb 04 14:34:21 BoxError: Backuptask crashed Feb 04 14:34:21 Node.js v20.18.0 Feb 04 14:34:21 } Feb 04 14:34:21 } Feb 04 14:34:21 }Thanks!
-
Backups keep failingNo new immich backup. Right now the app is marked as Error and on the backup tab the 'Create backup' button is greyed-out.
In the notifications I also get a backup task crashed notification.
Here are the logs for both immich and that general backup task crashed.https://upload.disroot.org/r/nuvYocg7#xGyeIOHzUekVeQhAC3Aup1vR7/SZqPepg3/aC10aSW0=
https://upload.disroot.org/r/9MFpVBdn#B9BzBZ8buWuNh0SPq6j605GBQcCqIedo0G/jI2J956g=Please have a look at the logs.
Thanks! -
Backups keep failingI am getting these errors now:
Jan 27 20:47:44 descriptor Jan 27 20:47:44 [01/27/25 19:47:44] INFO Error while closing socket [Errno 9] Bad file Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down Jan 27 20:47:44 [01/27/25 19:47:44] INFO Shutting down due to inactivity. Jan 27 20:47:44 [01/27/25 19:47:44] INFO Waiting for application shutdown. Jan 27 20:47:48 [01/27/25 19:47:48] ERROR Worker (pid:772) was sent SIGINT! Jan 27 20:47:48 [01/27/25 19:47:48] INFO Application shutdown complete. Jan 27 20:47:48 [01/27/25 19:47:48] INFO Booting worker with pid: 907 Jan 27 20:47:48 [01/27/25 19:47:48] INFO Finished server process [772] Jan 27 20:47:57 of inactivity. Jan 27 20:47:57 [01/27/25 19:47:57] INFO Application startup complete. Jan 27 20:47:57 [01/27/25 19:47:57] INFO Created in-memory cache with unloading after 300s Jan 27 20:47:57 [01/27/25 19:47:57] INFO Initialized request thread pool with 4 threads. Jan 27 20:47:57 [01/27/25 19:47:57] INFO Started server process [907] Jan 27 20:47:57 [01/27/25 19:47:57] INFO Waiting for application startup. Jan 28 01:00:00 2025-01-28T00:00:00Z Jan 28 01:00:00 2025-01-28T00:00:00Z Jan 28 01:00:00 2025-01-28T00:00:00Z Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Error: VipsJpeg: Premature end of input file Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object: Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object: Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Object: Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/66/b4/66b49043-0cc2-48a9-b163-c138a82362b8.HEIC Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: Input file is missing: upload/upload/4404f1b3-8534-432f-8c2e-ba205381eb5c/e4/e6/e4e69bc7-abab-4fb2-9b16-d01f28d3819d.jpg Jan 28 01:00:00 [Nest] 32 - 01/28/2025, 12:00:00 AM ERROR [Microservices:JobService] Unable to run job handler (thumbnailGeneration/generate-thumbnails): Error: VipsJpeg: Premature end of input file Jan 28 01:00:00 "id": "60e8f369-c07d-4e55-b54a-904f455f8366" Jan 28 01:00:00 "id": "78bd4060-73b1-4687-ad99-719856cfed7f"and
Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) Jan 28 01:00:00 at async Worker.retryIfFailed (/usr/src/app/node_modules/bullmq/dist/cjs/classes/worker.js:581:24) Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Jan 28 01:00:00 at process.processTicksAndRejections (node:internal/process/task_queues:95:5) Jan 28 01:00:00 { Jan 28 01:00:00 { Jan 28 01:00:00 { Jan 28 01:00:00 } Jan 28 01:00:00 } Jan 28 01:00:00 } Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:Backup failed with code 1] Database Backup Failure Jan 28 03:00:00 R [Microservices:BackupService] Backup failed with code 1 Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] Gzip exited with code 0 but pgdump exited with 1 Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM ERROR [Microservices:BackupService] pg_dumpall: error: query failed: ERROR: permission denied for table pg_authid Jan 28 03:00:00 [Nest] 32 - 01/28/2025, 2:00:00 AM LOG [Microservices:BackupService] Database Backup Starting. Database Version: 14 Jan 28 03:00:00 Jan 28 03:00:00 pg_dumpall: error: query was: SELECT rolname FROM pg_authid WHERE rolname !~ '^pg_' ORDER BY 1 Jan 28 16:22:36 [Nest] 47 - 01/28/2025, 3:22:36 PM LOG [Api:EventRepository] Websocket Connect: LBAslGNTdfqXxBm4AAAL Jan 28 16:23:39 [Nest] 47 - 01/28/2025, 3:23:39 PM LOG [Api:EventRepository] Websocket Disconnect: LBAslGNTdfqXxBm4AAAL Jan 28 18:31:37 ['CPUExecutionProvider'], in descending order of Jan 28 18:31:37 preference Jan 28 18:31:37 [01/28/25 17:31:37] INFO Loading visual model 'ViT-B-32__openai' to memory Jan 28 18:31:37 [01/28/25 17:31:37] INFO Setting execution providers to Jan 28 18:31:39 ['CPUExecutionProvider'], in descending order of Jan 28 18:31:39 preference Jan 28 18:31:39 [01/28/25 17:31:39] INFO Loading detection model 'buffalo_l' to memory Jan 28 18:31:39 [01/28/25 17:31:39] INFO Setting execution providers to Jan 28 18:31:40 ['CPUExecutionProvider'], in descending order of Jan 28 18:31:40 preference Jan 28 18:31:40 [01/28/25 17:31:40] INFO Loading recognition model 'buffalo_l' to memory Jan 28 18:31:40 [01/28/25 17:31:40] INFO Setting execution providers to Jan 28 18:36:39 descriptor Jan 28 18:36:39 [01/28/25 17:36:39] INFO Error while closing socket [Errno 9] Bad file Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down Jan 28 18:36:39 [01/28/25 17:36:39] INFO Shutting down due to inactivity. Jan 28 18:36:39 [01/28/25 17:36:39] INFO Waiting for application shutdown. Jan 28 18:36:40 [01/28/25 17:36:40] ERROR Worker (pid:907) was sent SIGINT! Jan 28 18:36:40 [01/28/25 17:36:40] INFO Application shutdown complete. Jan 28 18:36:40 [01/28/25 17:36:40] INFO Booting worker with pid: 946 Jan 28 18:36:40 [01/28/25 17:36:40] INFO Finished server process [907] Jan 28 18:36:47 of inactivity.I am not sure how to handle that...