🧪 Testers wanted: Element Server Suite (ESS Community) for Cloudron

[humpty]

Correct me if I'm wrong as I'm trying to wrap my head around this:

• I cannot install this app alongside an existing matrix/element on the same domain
• The bare domain must be empty (no app installed)
• The primary location must be "account" (exact wording) and not something else e.g. comms.domain.com
• Running this app alongside existing matrix app on different domains BUT on the same CR is OKAY

[andreasdueren]

@humpty said:

Correct me if I'm wrong as I'm trying to wrap my head around this:

• I cannot install this app alongside an existing matrix/element on the same domain

Correct, same with any other matrix/synapse… software thought. You can however install it on a subdomain. But it’s probably easier to just use a different domain.

• The bare domain must be empty (no app installed)

For federation to work and because of current well-known limitations with Cloudron, i recommend setting the tld as an alias on the EMS app after installation. If/when Cloudron enables customization of the well-known endpoints, this shouldn’t be an issue anymore. Currently it’s a workaround.

• The primary location must be "account" (exact wording) and not something else e.g. comms.domain.com

It can be anything. Could be ems.matrix.domain.com, there are no naming restrictions. I simply would have preferred to host the chat (element) app on this domain but for oidc reasons this needs to be Ems. Maybe I phrased this badly in the OP.

• Running this app alongside existing matrix app on different domains BUT on the same CR is OKAY

Correct you can install this on the same Cloudron as your existing installation and won’t interfere if you don’t override domains used for that instance.

[humpty]

@andreasdueren said:

Maybe I phrased this badly in the OP.

It's definitely me. I just switched from iOS to GrapheneOS and my tech-brain is fried. So many things that "just worked" in iOS requires a bridge, an extra app, or some hack to get to work. More details here

[andreasdueren]

@humpty yes I understand, I’ve made these experiences too. My goal for this Ems package is to just fix it all and make it just all work. Lol

[andreasdueren]

Update — use v0.3.13 or later

Do not use the migration implementation from v0.3.11 or v0.3.12.
A subsequent real-data end-to-end rehearsal found and fixed several issues,
including restoring PostgreSQL while Synapse/MAS could still be running,
incorrect syn2mas configuration, loss of MAS migration bookkeeping, and
missing appservice registrations.

The corrected workflow was published in v0.3.13 and tested using a
disposable ESS installation and a copy of a real Synapse database. The live
source homeserver was not modified.

Migration must be performed with the new ESS target in Cloudron
debug/recovery mode, with Synapse and MAS completely offline. Follow the
current docs/MIGRATION.md,
not the original v0.3.11 instructions below.

Migration from existing Synapse package now available in v0.3.11

I added a first guided migration path for moving an existing Synapse homeserver into the ESS Cloudron package.

This is experimental: Don't use in production unless you know the risks.

This is meant for people currently running the old/legacy Synapse package who want to move to the new ESS stack with Matrix Authentication Service (MAS), Element X, and Element Call, without asking every user to log in again.

The package now ships a helper script:

/app/pkg/src/migrate-from-synapse.sh  

It uses Element's official mas-cli syn2mas migration tooling under the hood.

What it preserves

The goal is a real homeserver migration, not a fresh start.

The migration is designed to preserve:

Data	Preserved?	Notes
Matrix server name		Must stay exactly the same
Users		Existing users are imported
Password hashes		Old Synapse bcrypt hashes continue working
Password pepper		Auto-detected from old homeserver.yaml if present
Existing sessions		Users should stay logged in
Devices		Existing clients/devices are imported
Access tokens		No forced logout expected
SSO/OIDC account links		Existing Synapse OIDC provider is mapped to MAS
Rooms / room history		Comes from the Synapse database
E2EE data		Comes from the Synapse database
Media store		Copied from the old media store
Signing key		Reused, so federation identity stays intact

In other words: the intention is that users do not need to take any action.

Important requirements

This is only for migrations where the Matrix server name stays identical.

For example, if your current users are:

@alice:example.com  
@bob:example.com  

then the ESS app must also be installed for server name:

example.com  

You cannot migrate example.com to matrix.example.com or to a different domain. Matrix server names are permanent.

Other requirements / caveats:

• SAML/LDAP/custom password providers are not supported by MAS directly.
• OIDC/SSO migrations should work, but the old Synapse OIDC provider must be mapped to the new MAS provider.
• The old and new apps cannot serve the same Matrix domains at the same time, so plan a short maintenance window for the final domain switch.
• This is still new — please test on a copy first.

How it works

The script expects migration files in:

/app/data/migration/  

Required files:

/app/data/migration/homeserver.yaml  
/app/data/migration/synapse.dump  
/app/data/migration/signing.key  

Optional but recommended:

/app/data/migration/media/  

Where:

File	Description
homeserver.yaml	The old Synapse config
synapse.dump	PostgreSQL dump of the old Synapse database, preferably pg_dump -Fc
signing.key	The old Synapse signing key
media/	The old Synapse media store

The script then:

1. Checks that the old and new server_name match.
2. Detects the old Synapse OIDC provider ID.
3. Detects Synapse password pepper if configured.
4. Takes a rollback snapshot of the new ESS app's current database/secrets.
5. Restores the old Synapse database into the new ESS app.
6. Copies the old signing key and media store.
7. Configures MAS to accept imported Synapse password hashes.
8. Configures MAS to map the old Synapse OIDC provider to the Cloudron OIDC provider.
9. Runs Element's official mas-cli syn2mas.
10. Leaves existing sessions/devices/access tokens imported into MAS.

Usage

Open a Web Terminal in the new ESS app and run:

/app/pkg/src/migrate-from-synapse.sh check  

This performs preflight checks only.

Then run:

/app/pkg/src/migrate-from-synapse.sh dry-run  

This performs a full rehearsal using syn2mas --dry-run.

The dry run is the important safety step. It should detect most problems before the real migration, including:

• server name mismatch
• unsupported password configuration
• unmapped SSO provider
• invalid database/config assumptions

If the dry run is clean, run the real migration:

/app/pkg/src/migrate-from-synapse.sh migrate  

The real migration requires typing:

MIGRATE  

as confirmation.

Afterwards, restart the app and switch the domains from the old Synapse/Element apps to the new ESS app.

Rollback / reverse migration

There are two safety mechanisms.

1. The old Synapse app is not touched

The migration reads from a database dump and copied files. It does not modify the old live Synapse app.

So before the final domain switch, the safest rollback is simply:

stop the ESS app → keep using the old Synapse app  

2. Script rollback for the ESS app

The script also creates a snapshot of the ESS app before replacing its database.

To restore that snapshot:

/app/pkg/src/migrate-from-synapse.sh rollback  

This requires typing:

ROLLBACK  

as confirmation.

This is useful if you want to retry a failed migration attempt from a clean ESS state.

However, Element's syn2mas migration itself is one-way. A true "MAS back to legacy Synapse auth after live use" migration does not exist upstream. Once users have actively used the migrated server, rolling back means losing anything that happened after the migration.

Example export from old Cloudron Synapse

In the old Synapse app's Web Terminal:

pg_dump -h "${CLOUDRON_POSTGRESQL_HOST}" -p "${CLOUDRON_POSTGRESQL_PORT}" \  
  -U "${CLOUDRON_POSTGRESQL_USERNAME}" -d "${CLOUDRON_POSTGRESQL_DATABASE}" \  
  -Fc -f /tmp/synapse.dump  

Then copy these into the new ESS app:

/tmp/synapse.dump  
/app/data/configs/homeserver.yaml  
the file referenced by signing_key_path  
the directory referenced by media_store_path  

into:

/app/data/migration/  

Status

I have tested the script flow end-to-end with stubbed database/MAS commands:

• check
• dry-run
• confirmation handling
• migrate
• double-migration guard
• rollback

What still needs real-world testing is the actual syn2mas import against different existing Synapse deployments. If you're running the old Synapse package and are willing to test on a copy, feedback would be extremely helpful.

[andreasdueren]

Migration update: v0.3.13 tested with real Synapse data

A real end-to-end migration rehearsal has now been completed using a
disposable ESS target and a copy of an existing PostgreSQL-backed Synapse
homeserver. The live source server was not modified.

The rehearsal covered:

• restoring a real Synapse PostgreSQL database;
• preserving the original Matrix server name and signing key;
• copying the media store;
• running the official mas-cli syn2mas dry run and real import;
• importing human users, password hashes, OIDC links, devices, sessions, and
  access tokens;
• verifying an imported access token against
  /_matrix/client/v3/account/whoami;
• preserving existing appservice registrations;
• retaining rooms, history, E2EE/device data, and media;
• restoring the clean ESS target through the rollback command.

The test uncovered several problems in the original v0.3.11 workflow. These
included ineffective process shutdown, the wrong Synapse configuration being
passed to syn2mas, MAS migration metadata being removed during database
restore, warning exit codes being treated as failures, and appservice
registrations not being copied.

These issues are fixed in v0.3.13.

Do not use v0.3.11 or v0.3.12 for migration. Use v0.3.13 or later.

The corrected process requires putting only the new ESS target into Cloudron
debug/recovery mode so Synapse and MAS are completely offline before the
database is restored. The source Synapse installation remains untouched and
continues to be the fallback until the final domain cutover.

Current instructions:

https://git.due.ren/andreas/cloudron-ess/-/blob/main/docs/MIGRATION.md

Although the workflow has now passed a real-data rehearsal, migration should
still be treated as a maintenance operation: take backups, rehearse with a
copy, inspect the syn2mas warnings, and plan the domain cutover carefully.

[humpty]

I set up a new server. The installation went smoothly. I logged in via Cloudron using two different CR users.

User 1: logged in to element x on iOS
User 2: logged in to element x on GrapheneOS/Android (installed thru google play store)

iOS to Android:

• Text messages
• Audio calls
• Video calls

I'll test iOS to iOS when I get my hands on fam devices tonight.

Edit: BTW, I tried the option "blur background" in the video settings and things became REALLY laggy on my Pixel 10. I enabled it during the call. Maybe that's why.

[andreasdueren]

Migration update: guided one-command Synapse → ESS wizard in v0.3.17

The migration workflow has now been wrapped in an interactive wizard. After installing a fresh ESS target on temporary domains, run this on a Mac or Linux computer where the Cloudron CLI is installed and logged in:

bash <(curl -fsSL https://git.due.ren/andreas/cloudron-ess/-/raw/v0.3.17/scripts/cloudron-synapse-to-ess.sh)

No migration arguments are required. The wizard asks for:

1. the domain of the old Cloudron Synapse app;
2. the primary domain of the fresh ESS target;
3. rehearsal or final migration;
4. explicit confirmation before starting;
5. a second confirmation after the complete syn2mas dry-run.

In final mode it freezes the old homeserver before the database dump, leaves both apps safely offline for the domain switch, and starts ESS only after the operator confirms CUTOVER-DONE.

The wizard handles Cloudron backups, PostgreSQL export/restore, the original signing key and server name, appservice registrations, local media, S3 configuration, target recovery mode, preflight checks, the official syn2mas dry-run/import, and rollback.

One important media detail: local media is copied even when S3 is enabled. S3 may have been enabled after older media was created, or an asynchronous upload may be incomplete. The migrated target therefore preserves the union of all local files and existing S3 objects instead of assuming the bucket is complete.

I completed a real end-to-end rehearsal through the new no-argument terminal UI using a fresh disposable ESS target and an existing Cloudron Synapse database. It completed with exit code 0 and verified:

• 3,061 Synapse user rows and all 45 human users;
• matching imported human compatibility sessions;
• 29,533 local media files / 2,389,933,963 bytes;
• identical signing-key and S3-provider fingerprints;
• identical contents for both appservice registrations;
• successful read access to the existing S3 bucket;
• removal of credential-bearing staging data after success.

The old homeserver stayed online throughout the rehearsal. The disposable target was kept in recovery mode, verified, then uninstalled. Only backups created by the test were marked for immediate expiry, and they have already been cleaned up.

Documentation:

• Guided wizard: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.17/scripts/cloudron-synapse-to-ess.md
• Full migration reference/manual fallback: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.17/docs/MIGRATION.md

The final production domain reassignment remains an explicit Cloudron dashboard step because legacy Synapse/Element domains and bare-domain well-known setups differ between installations. The wizard does not guess or silently take over domains.

[andreasdueren]

Migration update: full production-style cutover tested end-to-end — v0.3.18

Since the last update I ran the complete final-migration path (not just a rehearsal) on a dedicated test Cloudron, exactly the way a real production switch would go.

Test setup

• Fresh legacy Cloudron Synapse (org.matrix.synapse 1.138.1) as the source
• Two users, both linked to Cloudron SSO
• Five live device sessions before migration: Element X on iOS, Element X on Android, and Element Web/Safari sessions
• Fresh ESS target installed on temporary domains via the community catalog

What the wizard did

Rehearsal first (validated, target discarded), then final mode: backups of both apps, source frozen before the export, PostgreSQL/signing-key/config transfer, syn2mas dry-run, real import, then the interactive CUTOVER-DONE gate while I moved the production domains over in the dashboard.

The result

All five pre-migration device sessions survived the migration without re-authentication. Both phones and the browser sessions just kept syncing against the migrated MAS+Synapse — token sets verified hash-identical, device IDs preserved in MAS compatibility sessions, signing key identical, SSO links and password hashes intact. Calls work too (after the fix below).

Two things the test uncovered, both addressed in v0.3.18

1. Wizard bug (fixed): the post-cutover health check probed the Matrix client API on the MAS port instead of Synapse's port, so a perfectly healthy cutover was reported as failed and the wizard (correctly, but unnecessarily) froze everything again. Failing closed instead of open was the designed behavior, but the probe itself was wrong.

2. Cutover pitfall (now documented in the wizard and docs): if your legacy setup used Cloudron's Domains → Well-Known feature for Matrix, those entries survive the migration and keep intercepting https://<server-name>/.well-known/matrix/client at the platform nginx — before ESS's apex alias ever gets asked. Since that fixed template can't carry rtc_foci, Element Call fails with MISSING_MATRIX_RTC_TRANSPORT even though ESS itself is completely healthy. Delete those dashboard well-known entries during cutover and let the ESS apex alias serve its own complete payload. Mobile Element X caches well-known at login, so force-close and reopen the app afterward.

Migration command (updated pin):

bash <(curl -fsSL https://git.due.ren/andreas/cloudron-ess/-/raw/v0.3.18/scripts/cloudron-synapse-to-ess.sh)

Docs:

• Guided wizard: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.18/scripts/cloudron-synapse-to-ess.md
• Migration reference / manual fallback: https://git.due.ren/andreas/cloudron-ess/-/blob/v0.3.18/docs/MIGRATION.md

Where this stands

With rehearsal, final migration, real domain cutover, session survival, SSO, and calls all verified end-to-end, I consider this essentially production-ready. I'm mainly waiting for feedback from more testers with different setups (bridges/appservices, large media stores, S3, federation-heavy servers) before calling it stable — and my own plan is to migrate my production Matrix servers over soon.

If you test it, please report back either way — successful boring migrations are just as useful as bug reports.

[humpty]

@andreasdueren Does the migration process work on existing installs to keep using the same domain?

[andreasdueren]

@humpty yes that’s what it’s designed to do

[andreasdueren]

Pinging the Cloudron team @girish @nebulon one more time more cleanly for this, or a comparative feature request (if you give me permissions on git.cloudron.io I’m happy to commit something directly):

andreasdueren said:

Request for the Cloudron team

@girish @nebulon The apex-domain situation (limitation #1) could be solved cleanly with a small platform feature: let an app declare custom well-known entries for its parent/apex domain — either in CloudronManifest.json or via an API the app can call — so Cloudron's nginx serves e.g. https://example.com/.well-known/matrix/{server,client} and /.well-known/element/element.json on the app's behalf, without the app claiming the whole apex as an alias.

Cloudron already serves some well-known content per domain, but there's no way to set the arbitrary JSON that Matrix needs today (e.g. the org.matrix.msc4143.rtc_foci transport entry that Element Call requires), and no way for a package to register it automatically. A wellKnown section in the manifest (updated on configure/restart) would benefit any protocol using apex-domain discovery — Matrix, WebFinger/Mastodon delegation, etc. Happy to write up a concrete proposal or beta-test an implementation with this package.

[girish]

@andreasdueren is this same or different from https://docs.cloudron.io/domains#matrix-server-location ? I guess we can also add element.json there.

[andreasdueren]

@girish Almost the same, with two differences:

1. That feature only serves a fixed set of locations (matrix/server, matrix/client, host-meta, caldav/carddav). element/element.json isn't in the whitelist, so adding it to the domain config today would still 404. If you're open to adding it there, or better, serving any configured location, that solves it. I actually have a small patch for the generic version ready (branch wellknown-generic-locations on https://git.due.ren/andreas/cloudron-box, incl. regression test) if you want an MR.

2. The entries are static text the admin maintains by hand, while the correct values (e.g. rtc_foci for Element Call, the MAS auth block) come from the app and change with its config. That's the angle of my feature request: letting an app declare/update these entries for its apex domain itself, so they can't go stale. The dashboard feature would stay as the manual override.

So: same mechanism, just asking for it to be un-whitelisted and optionally app-populated.

[james]

fyi: @andreasdueren
https://git.due.ren/andreas/cloudron-box can't be accessed without an account.

[andreasdueren]

@James Oops, sorry about that. It's public now. https://git.due.ren/andreas/cloudron-box/-/commit/314e5f7a39aec36e61bc21dfced5f26dec797b96

[girish]

We can take a look at this after Cloudron 10, thanks.