I can recommend to disable e2ee for channels. This makes everything smoother and especially allows all clients to search for messages without downloading everything. If you trust the server (probably your own) with the synapse instance, e2ee doesn't bring much extra security wise.