🧪 Testers wanted: Element Server Suite (ESS Community) for Cloudron

[andreasdueren]

Migration update: v0.3.13 tested with real Synapse data

A real end-to-end migration rehearsal has now been completed using a
disposable ESS target and a copy of an existing PostgreSQL-backed Synapse
homeserver. The live source server was not modified.

The rehearsal covered:

• restoring a real Synapse PostgreSQL database;
• preserving the original Matrix server name and signing key;
• copying the media store;
• running the official mas-cli syn2mas dry run and real import;
• importing human users, password hashes, OIDC links, devices, sessions, and
  access tokens;
• verifying an imported access token against
  /_matrix/client/v3/account/whoami;
• preserving existing appservice registrations;
• retaining rooms, history, E2EE/device data, and media;
• restoring the clean ESS target through the rollback command.

The test uncovered several problems in the original v0.3.11 workflow. These
included ineffective process shutdown, the wrong Synapse configuration being
passed to syn2mas, MAS migration metadata being removed during database
restore, warning exit codes being treated as failures, and appservice
registrations not being copied.

These issues are fixed in v0.3.13.

Do not use v0.3.11 or v0.3.12 for migration. Use v0.3.13 or later.

The corrected process requires putting only the new ESS target into Cloudron
debug/recovery mode so Synapse and MAS are completely offline before the
database is restored. The source Synapse installation remains untouched and
continues to be the fallback until the final domain cutover.

Current instructions:

https://git.due.ren/andreas/cloudron-ess/-/blob/main/docs/MIGRATION.md

Although the workflow has now passed a real-data rehearsal, migration should
still be treated as a maintenance operation: take backups, rehearse with a
copy, inspect the syn2mas warnings, and plan the domain cutover carefully.

[humpty]

I set up a new server. The installation went smoothly. I logged in via Cloudron using two different CR users.

User 1: logged in to element x on iOS
User 2: logged in to element x on GrapheneOS/Android (installed thru google play store)

iOS to Android:

• Text messages
• Audio calls
• Video calls

I'll test iOS to iOS when I get my hands on fam devices tonight.

Edit: BTW, I tried the option "blur background" in the video settings and things became REALLY laggy on my Pixel 10. I enabled it during the call. Maybe that's why.

[andreasdueren]

Migration update: guided one-command Synapse → ESS wizard in v0.3.17

The migration workflow has now been wrapped in an interactive wizard. After installing a fresh ESS target on temporary domains, run this on a Mac or Linux computer where the Cloudron CLI is installed and logged in:

bash <(curl -fsSL https://git.due.ren/andreas/cloudron-ess/-/raw/v0.3.17/scripts/cloudron-synapse-to-ess.sh)

No migration arguments are required. The wizard asks for:

1. the domain of the old Cloudron Synapse app;
2. the primary domain of the fresh ESS target;
3. rehearsal or final migration;
4. explicit confirmation before starting;
5. a second confirmation after the complete syn2mas dry-run.

In final mode it freezes the old homeserver before the database dump, leaves both apps safely offline for the domain switch, and starts ESS only after the operator confirms CUTOVER-DONE.

The wizard handles Cloudron backups, PostgreSQL export/restore, the original signing key and server name, appservice registrations, local media, S3 configuration, target recovery mode, preflight checks, the official syn2mas dry-run/import, and rollback.

One important media detail: local media is copied even when S3 is enabled. S3 may have been enabled after older media was created, or an asynchronous upload may be incomplete. The migrated target therefore preserves the union of all local files and existing S3 objects instead of assuming the bucket is complete.

I completed a real end-to-end rehearsal through the new no-argument terminal UI using a fresh disposable ESS target and an existing Cloudron Synapse database. It completed with exit code 0 and verified:

• 3,061 Synapse user rows and all 45 human users;
• matching imported human compatibility sessions;
• 29,533 local media files / 2,389,933,963 bytes;
• identical signing-key and S3-provider fingerprints;
• identical contents for both appservice registrations;
• successful read access to the existing S3 bucket;
• removal of credential-bearing staging data after success.

The old homeserver stayed online throughout the rehearsal. The disposable target was kept in recovery mode, verified, then uninstalled. Only backups created by the test were marked for immediate expiry, and they have already been cleaned up.

Documentation:

• Guided wizard: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.17/scripts/cloudron-synapse-to-ess.md
• Full migration reference/manual fallback: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.17/docs/MIGRATION.md

The final production domain reassignment remains an explicit Cloudron dashboard step because legacy Synapse/Element domains and bare-domain well-known setups differ between installations. The wizard does not guess or silently take over domains.

[andreasdueren]

Migration update: full production-style cutover tested end-to-end — v0.3.18

Since the last update I ran the complete final-migration path (not just a rehearsal) on a dedicated test Cloudron, exactly the way a real production switch would go.

Test setup

• Fresh legacy Cloudron Synapse (org.matrix.synapse 1.138.1) as the source
• Two users, both linked to Cloudron SSO
• Five live device sessions before migration: Element X on iOS, Element X on Android, and Element Web/Safari sessions
• Fresh ESS target installed on temporary domains via the community catalog

What the wizard did

Rehearsal first (validated, target discarded), then final mode: backups of both apps, source frozen before the export, PostgreSQL/signing-key/config transfer, syn2mas dry-run, real import, then the interactive CUTOVER-DONE gate while I moved the production domains over in the dashboard.

The result

All five pre-migration device sessions survived the migration without re-authentication. Both phones and the browser sessions just kept syncing against the migrated MAS+Synapse — token sets verified hash-identical, device IDs preserved in MAS compatibility sessions, signing key identical, SSO links and password hashes intact. Calls work too (after the fix below).

Two things the test uncovered, both addressed in v0.3.18

1. Wizard bug (fixed): the post-cutover health check probed the Matrix client API on the MAS port instead of Synapse's port, so a perfectly healthy cutover was reported as failed and the wizard (correctly, but unnecessarily) froze everything again. Failing closed instead of open was the designed behavior, but the probe itself was wrong.

2. Cutover pitfall (now documented in the wizard and docs): if your legacy setup used Cloudron's Domains → Well-Known feature for Matrix, those entries survive the migration and keep intercepting https://<server-name>/.well-known/matrix/client at the platform nginx — before ESS's apex alias ever gets asked. Since that fixed template can't carry rtc_foci, Element Call fails with MISSING_MATRIX_RTC_TRANSPORT even though ESS itself is completely healthy. Delete those dashboard well-known entries during cutover and let the ESS apex alias serve its own complete payload. Mobile Element X caches well-known at login, so force-close and reopen the app afterward.

Migration command (updated pin):

bash <(curl -fsSL https://git.due.ren/andreas/cloudron-ess/-/raw/v0.3.18/scripts/cloudron-synapse-to-ess.sh)

Docs:

• Guided wizard: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.18/scripts/cloudron-synapse-to-ess.md
• Migration reference / manual fallback: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.18/docs/MIGRATION.md

Where this stands

With rehearsal, final migration, real domain cutover, session survival, SSO, and calls all verified end-to-end, I consider this essentially production-ready. I'm mainly waiting for feedback from more testers with different setups (bridges/appservices, large media stores, S3, federation-heavy servers) before calling it stable — and my own plan is to migrate my production Matrix servers over soon.

If you test it, please report back either way — successful boring migrations are just as useful as bug reports.

[humpty]

@andreasdueren Does the migration process work on existing installs to keep using the same domain?

[andreasdueren]

@humpty yes that’s what it’s designed to do

[andreasdueren]

Pinging the Cloudron team @girish @nebulon one more time more cleanly for this, or a comparative feature request (if you give me permissions on git.cloudron.io I’m happy to commit something directly):

andreasdueren said:

Request for the Cloudron team

@girish @nebulon The apex-domain situation (limitation #1) could be solved cleanly with a small platform feature: let an app declare custom well-known entries for its parent/apex domain — either in CloudronManifest.json or via an API the app can call — so Cloudron's nginx serves e.g. https://example.com/.well-known/matrix/{server,client} and /.well-known/element/element.json on the app's behalf, without the app claiming the whole apex as an alias.

Cloudron already serves some well-known content per domain, but there's no way to set the arbitrary JSON that Matrix needs today (e.g. the org.matrix.msc4143.rtc_foci transport entry that Element Call requires), and no way for a package to register it automatically. A wellKnown section in the manifest (updated on configure/restart) would benefit any protocol using apex-domain discovery — Matrix, WebFinger/Mastodon delegation, etc. Happy to write up a concrete proposal or beta-test an implementation with this package.

[girish]

@andreasdueren is this same or different from https://docs.cloudron.io/domains#matrix-server-location ? I guess we can also add element.json there.

[andreasdueren]

@girish Almost the same, with two differences:

1. That feature only serves a fixed set of locations (matrix/server, matrix/client, host-meta, caldav/carddav). element/element.json isn't in the whitelist, so adding it to the domain config today would still 404. If you're open to adding it there, or better, serving any configured location, that solves it. I actually have a small patch for the generic version ready (branch wellknown-generic-locations on https://git.due.ren/andreas/cloudron-box, incl. regression test) if you want an MR.

2. The entries are static text the admin maintains by hand, while the correct values (e.g. rtc_foci for Element Call, the MAS auth block) come from the app and change with its config. That's the angle of my feature request: letting an app declare/update these entries for its apex domain itself, so they can't go stale. The dashboard feature would stay as the manual override.

So: same mechanism, just asking for it to be un-whitelisted and optionally app-populated.

[james]

fyi: @andreasdueren
https://git.due.ren/andreas/cloudron-box can't be accessed without an account.

[andreasdueren]

@James Oops, sorry about that. It's public now. https://git.due.ren/andreas/cloudron-box/-/commit/314e5f7a39aec36e61bc21dfced5f26dec797b96

[girish]

We can take a look at this after Cloudron 10, thanks.