Cloudron Forum

To give an update, docs are mostly updated. 10.0.5 has been published. There is now a Ubuntu 26 migration guide (but only follow this after updating to 10.0.5 !)

I have now measured the whole thing on IONOS with Cloudron 10.0.4, and the cause is clear. It is on the IONOS side, but Cloudron hides the part that shows it. Short version: on IONOS, overwriting (and deleting) an object costs about 24 seconds per GiB, handled synchronously before the response. Above roughly 25 GiB that exceeds the 600 s timeout of the proxy in front of their S3 service. The AWS SDK then silently repeats the CompleteMultipartUpload, one attempt answers 200 with an empty ETag, and afterwards the object is listed but its data can not be read. The rotate copy then fails with exactly the NoSuchKey from the first post. @dsp76, three things would help: In the log of a failed backup task: is there a long gap (around 10, 20 or 30 minutes) between the last "Uploading backup …" line and "upload completed"? How large are the other apps' snapshots? By these measurements, the problem should only affect apps above roughly 25 GiB, which would explain why only your largest app fails. Can the first MiB of the current snapshot still be downloaded, or does that return NoSuchKey? For example: aws s3api get-object --bucket <bucket> --key <prefix>/snapshot/app_c08ad55d-….tar.gz.enc --range bytes=0-1048575 --endpoint-url <your IONOS endpoint> /tmp/snapshot-test.bin Below is a separate summary for IONOS support with all request IDs. For IONOS support (all times UTC, 2026-09-11) Hello IONOS team, we have measured this with Cloudron's own S3 code and logged every request with x-amz-request-id. There are two separate issues: the runtime of an overwrite, and an object that is confirmed but unreadable afterwards. Client and test setup Endpoint https://s3.eu-central-3.ionoscloud.com, region de, virtual-hosted-style requests, AWS SDK for JavaScript v3 (@aws-sdk/client-s3 3.1116.0), Cloudron 10.0.4, one client host Buckets repro-15914, repro-15914-b, repro-15914-c, repro-15914-d; objects of random data uploaded as multipart, 3 parts in parallel 1. CompleteMultipartUpload of an overwrite scales with the size of the object being replaced Case CompleteMultipartUpload Key does not exist yet (create), 0.1 GiB to 39 GiB under 1 s in every case Key already exists (overwrite) about 23.5 s per GiB of the old object DeleteObject on such an object about 25 s per GiB Independent of the part size (50, 100 and 250 MiB parts at the same size give the same time) and of the bucket. Reproducible cheaply: create a 3.2 GiB object and overwrite it, which takes about 75 s instead of about 0.2 s. Occasionally, in 3 of about 34 operations, an overwrite or delete was fast (0.2–0.5 s). Question: why is this work done synchronously before the response, and what happens with it after 600 s? A proxy in front of the service answers with 504 Gateway Time-out (HTML, no S3 headers, no request ID) after exactly 600.0 s. For comparison, AWS S3 sends the 200 header early for long-running completions and keeps the connection alive with whitespace. source: https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html 2. Above 600 s the object ends up unreadable Bucket repro-15914-d, key repro-15914/snapshot/app_p2-s40000-r1.tar.gz.enc, 41,943,040,000 bytes in 800 parts, upload 2~C7vvA9XH1m0gX5ETzPptO6piVDoqJKw: Sent Request Response x-amz-request-id 22:16:20.100 CompleteMultipartUpload, attempt 1 504 after 600.07 s none 22:26:40.176 CompleteMultipartUpload, attempt 2 504 after 600.06 s none 22:37:00.241 CompleteMultipartUpload, attempt 3 200 in 78 ms, body contains <ETag></ETag> tx00000898b3e6a0332a144-006aa4828c-810022444-eu-central-3 22:37:00.328 HeadObject 200, 41943040000 bytes, ETag "d6198b07dbf3a8ecd1f840d8b24c7c35-800", Last-Modified 22:26:40 tx0000019e93fb95908b459-006aa4828c-809736856-eu-central-3 22:37:00.478 UploadPartCopy part 1, bytes=0-1073741823 404 NoSuchKey, empty <Message> tx000009e4ad2fa8b99db77-006aa4828c-809736856-eu-central-3 22:37:00.483 UploadPartCopy part 2 404 NoSuchKey tx00000fe72582a902f5490-006aa4828c-807519665-eu-central-3 22:37:00.530 UploadPartCopy part 3 404 NoSuchUpload (our client had already aborted the destination upload) tx00000d35fc6d29c7099a6-006aa4828c-809730471-eu-central-3 22:37:01.068 GET bytes=0-1048575 404 NoSuchKey tx00000849e27e44423877a-006aa4828d-797521041-eu-central-3 22:37:01.110 GET bytes=20971520000-20972568575 404 NoSuchKey tx000002052bd3daec1d0ba-006aa4828d-809730471-eu-central-3 22:37:01.197 GET bytes=41941991424-41943039999 206, 1048576 bytes tx00000ad11ad4192c75825-006aa4828d-803482478-eu-central-3 22:40:31 the same three ranges again 404 / 404 / 206, unchanged tx0000095e483f0c394dbf5-…, tx0000017020a5bb1e46bd8-…, tx0000018734af690b0297b-… Control in the same bucket: repro-15914/repro/2026-09-11T21-50-54-233Z/app_repro-15914_v1.tar.gz.enc (39 GiB, created by a server-side copy) returns 206 for all three ranges at 22:40:31. The same happened earlier in bucket repro-15914 with a 72.6 GB object: three 504s, then 200 with an empty ETag, and after that all three ranges returned NoSuchKey, still 33 minutes later. That object is still in the bucket for you to look at, key repro-15914/snapshot/app_repro-15914.tar.gz.enc, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", along with an intact control object. Just to make sure this is not a timing issue and the large object might now readable: All three ranges of the first manual object return NoSuchKey: Range Result x-amz-request-id bytes=0-1048575 404 NoSuchKey tx00000760487e9eeddc8f0-006aa4a32b-809730471-eu-central-3 bytes=36304322560-36305371135 404 NoSuchKey tx000008f860d7695a5e683-006aa4a32b-803482478-eu-central-3 bytes=72607596544-72608645119 404 NoSuchKey tx0000037d2d2c104ca6b58-006aa4a32b-803482478-eu-central-3 HeadObject still claims the object is fine: HTTP 200, 72,608,645,120 bytes, ETag "252d72dfd55e5f6b8799cb56a64339b3-1385", Last-Modified 14:49:28, identical to 10 hours ago (tx0000095cf0eba4266169e-006aa4a32b-799478461-eu-central-3). Questions Why does CompleteMultipartUpload of an overwrite take about 24 s per GiB, and is the 504 produced by a proxy with a 600 s timeout? Attempt 3 answered 200 with an empty ETag. Which request actually completed the upload? The object's Last-Modified matches attempt 2, which received a 504. Why is the object listed and returned by HeadObject, while GET and UploadPartCopy answer NoSuchKey on several gateway hosts, unchanged minutes later? Is the data lost, and can it be recovered? Is the empty ETag in the CompleteMultipartUpload response a reliable marker on your side that the object was not stored correctly? You mentioned requests referring to an upload ID that no longer exists. In our logs the only upload IDs sent more than once belong to the repeated CompleteMultipartUpload attempts; the part 3 NoSuchUpload above comes from our own abort. Could those be the requests you found? The affected objects are kept in repro-15914 and repro-15914-d. If needed, I can provide the full log files of each attempt of reproducing this.

A YT guy always ends his prompts with ".., answer in short."

@girish said: Fixed. We display inactive users separately now at the bottom of the users list. What does "inactive" mean here? Edit: Ah, presumably it means this box in the edit user view is unchecked: [image: image.jpeg]

[1.28.0] Update sockpuppetbrowser to a4a9ee3

@andreasdueren Hi again! I updated to the latest version (0.4.89) and I can confirm that HERMES_REDACT_SECRETS is successfully read from scheduled cron jobs! Thanks for the fix! Now I have a new but similar issue issue with GITHUB_TOKEN - it is present in the .env (agent confirms that in the chat), but it's not being read in the cron. Here is the report from the agent: The cron scheduler runs inside the gateway process. When it spawns the no_agent script, it builds the child env via build_subprocess_env() → _sanitize_subprocess_env() which unconditionally strips GITHUB_TOKEN / GH_TOKEN (they're in _ALWAYS_STRIP_KEYS in tools/environments/local.py). The token is present in /app/data/.hermes/.env (line 436, uncommented, len 93) and in my interactive shell, but it never reaches the cron script's subprocess. This is a design conflict: the backup runner needs the token, but Hermes' secret-scrub policy deliberately removes it from child processes. GITHUB_TOKEN is not registered in config.yaml terminal.env_passthrough, so the scrub isn't bypassed. Proposed fix (from our side, not yours) A small wrapper-only code change in our python script that uses github for backup – not a config.yaml change and not a gateway restart. Current flow: cron scheduler (sanitized env; token stripped) → github_backup_publish.py → run_github_backup.py --publish Proposed flow: cron scheduler (sanitized env) → github_backup_publish.py → read /app/data/.hermes/.env internally → extract only GITHUB_TOKEN → start run_github_backup.py with the existing sanitized env + only GITHUB_TOKEN After I did this workaround, the github token is successfully read. My question for you is this something you could also fix, or I should keep this (or some better) workaround? Thank you once again for your time!

I got the same as @sponch did earlier. Getting the same after the latest version. Also FYI UDP port 3478 cannot be used (reserved). Ticking the STUN NAT Traversal box and setting and saving 3479 superficially works, but on going back to 'Location' in app portal, 3479 has not been saved, it's reverted to 3478 and the tick box to enable custom port has been unticked. I don't think the app CloudronManifest.json is using the cloudron turn addon, which might (not sure) allow port 3478 to be used.

Thanks for packaging Bitwarden Lite. I am personally not particularly interested in it. However having an alternative to Vaulwarden on Cloudron would be appreciated. Possibly, something like Psono, AliasVault or Passbolt as a suggestion if you needed some But for now, thanks again.

Another thing if you activate codebase memmory on repos the search with graphql is instant and sometimes I thought models are hallucinating even Fable 5.1. But it wasn’t the case, whole repo loaded and schemas loaded in ram and results are at least 4 times faster then normal windows based same tooling and usage as no overhead of windows filesystem nor Microsoft defender