Cloudron Forum

@murgero EteSync is CardDAV compliant and SOGo is not. I've had trouble making SOGo connect with Apple Contacts on both macOS 14 and iOS 13. EteSync is working for me.

Initial setup of EteSync is more painful but:

1. in the end it works
2. you get properly encrypted data on their server, with no secret master keys with the provider (open source, we can check)
3. European company (French I believe) so there is a hope they walk the talk about privacy (US companies by law are not allowed to provide privacy).

The price of admission is $24/year which is fine, as one doesn't have to run the server. There's an unencrypted at rest CardDAV subuser included which will allow people who care about privacy to share an address book or calendar with people who don't (looking at most of our girlfriends/partners here, although I'm sure there are some extremely privacy conscious ladies who are frustrated with their mainstream who-cares-about-privacy-i-don't-have-anything-to-hide partners).

EteSync is running successfully for me and syncing between macOS and iOS 13 right now but there's a lot more testing to do before I can be sure that it's as good as billed. In macOS Contacts the original groups even turn up and work as they should. In iOS Contacts there is only one group "My Contacts".

Some long term users feel very positive about EteSync. Aerion asked ProtonMail to include it:

EteSync provide a ready to use, open source, and well established solution for adding end-to-end encryption: Etebase (https://www.etebase.com/). It already works with Android, iOS, Windows, Linux, macOS, Thunderbird, Evolution, Kontact, Tasks.org, and any other CalDAV/CardDAV client.
 
I've been using it for a number of years already, and it's never failed, not once. It's the only sync solution I've ever used that's not had problems with duplicates or missing data.

Reliable sync with no data loss is one of the hardest of software tasks. EteSync for now managed to retain groups in iOS/macOS which none of the other CardDAV calendars I've tried have managed.

@luckow Thanks for the tip about enforcing use through Thunderbird/a compliant client. Any idea of how SOGo fares with iOS sync?

@jdaviescoates Thanks for the tip about the Sabre/dav request. I've added my feedback (not positive sadly) based on five years of active use of Fruux with various MacOS and recent version of iOS (I'm a relative latecomer to iPhones as I stopped using cellphones between about 2013 and 2018: now I have to have one for banking unfortunately and am surrounded by a pack of rabid iPhone users both at work and at home).

PS. Could someone somewhere please upvote a post for me so I can post more often than every six minutes.

@drew No, I haven't and I'm very frustrated. Fruux CardDAV doesn't sync with anything external any more at all (the calendards work). Website only at least for modifications. NextCloud can work but is almost a full time IT project to make NextCloud perform reliably. Experimenting with EteSync now. It's hosted on someone else's server but is open source and driven to work with lots of open source projects.

Doesn't help that Apple Contacts is back working okay with CardDAV in Monterrey (was terrible for years post-Snow Leopard) but BusyContacts won't sync reliably with CardDAV any more (even after paying for the expensive update – if this persists I'll be refunding the update and cutting ties with BusyMac – yet again, as I was burned by the same developers with Now Up-to-Date and Now Contacts which worked until they didn't).

The whole simple CardDAV/CalDAV idea has been effectively embrace-extend-extinguished by Apple. Seems like a simple concept to start with. How could 150,000 development man hours poured into calendar and contact sync end up with only working software on Google or Apple? Argh.

@drew It's early innings but the Cloudron SoGo installation looks like a winner. I have no intention of using SoGo as a primary email mailbox as we'd quickly bring our Cloudron server to its knees with 150 GB of mail across the company but the CalDAV and the CardDAV functionality appears to work well.

Here's some good end user guides for how to use SoGo, written by a Swedish ISP.

• How to manage address book in SOGo webmail
• How to manage calendar in SOGo webmail

The guides include how to share an address book which is obvious once you know how but not obvious until then.

Interesting results so far with SoGo. Won't work with Apple Contacts 2498.2.1 on macOS 14, nor on iPhone Contacts no matter what handstands I do with the custom paths or not.

/SOGo/dav/yourusername/Contacts/personal/

I am able to connect with both CardDAV and LDAP with Apple Contacts on macOS but none of the existing contacts show up, nor is it possible to add new cards to either the CardDAV or LDAP version. Hallucinating.

But SoGo does work on BusyContacts just fine picking up both address books and saving successfully back to the web version.

There is no iOS version of BusyContacts and I don't want to tie myself down to a third party application to be able to use shared address books. The shared address book is supposed to work between my partner and I and later within the company, which means a server which will work with Apple Contacts.

Why, or why can't someone code a CardDAV/CalDAV solution which is simple to configure and which works reliably with default Apple clients? Fruux did seem to be the solution until they abandoned the project.

To follow up, some applications are just a pain-in-the-neck and are difficult to make work. For instance managing file space on NextCloud. NextCloud never wants to release files and delete them and it's very easy to run out of space and very hard to clear free space. These are complex issues and there are no simple solutions (apart from not running NextCloud which looks like it's probably our next step as the amount of time which goes into admin of NextCloud makes it a losing proposition). @girish was incredibly helpful and at least got us back to the starting line.

But in the case of InvoiceNinja, the application works, the preferences work. What was breaking InvoiceNinja was the Cloudron setup, not InvoiceNinja. Cloudron is effectively sabotaging effective work with an important app which does have its act together.

@marcusquinn Marcus, I noticed your crowing about NextCloud talk higher in the thread. It caught my eye. My own experience with NextCloud file sharing and address sync has been painful enough over the course of years (partly Apple's fault) that I'm not keen to rely more on NextCloud. It's been a high maintenance slog. Probably for hundreds of users it's worth it but certainly not for half a dozen users.

To be honest, until relatively recently our RocketChat experience has been much better. We were early users and really loved this app, it's just become so community unfriendly.

@LoudLemur I'll take a look at Zulip. Thanks for the suggestion.

We are currently Fruux users but with enough issues with Fruux that we'd like to move our shared calendars and address books to a Cloudron app. We're currently using invoiceninja and rocketchat in production via Cloudron.

Should we go with SOGo or NextCloud? NextCloud has always seemed a bit top-heavy – we don't need to move email in and we are handling file storage in project management.

Radicale won't work for us as there is no well-thought-out permissions structure for shared calendars and address books which is important for us.

Anyway, we are deciding between SOGo or NextCloud for CalDAV/CardDAV sharing. Anyone here have a strong opinion after working with both?

Dark horse candidates also welcome. It's amazing that CardDAV and CalDAV are so difficult after the main issues were solved 15 years ago (both actually worked great with OS X 10.6 Snow Leopard. A clear case of Embrace, Extend, Extinguish by all of Apple, Microsoft, Google.

Hi Guys,

We've had deliverability issues for our Cloudron domain ourdomain.net and have moved to using SendGrid.com via ourdomain.com. I've set up SPF and DMARC successfully but there's no DKIM key.

ourdomain.com does have valid DKIM keys.

What this means concretely is:

billing.ourdomain.net sends its notifications via billing@ourdomain.com

Any suggestions on how to add working DKIM for ourdomain.net?

A related question is that I hope Cloudron will not wipe out the manually edited SPF records I've added directly in the DigitalOcean networking panel: https://cloud.digitalocean.com/networking/domains/ourdomain.net

For now, I'm sending without DKIM keys but would very much like to add them.

Thanks in advance for any expert help.

Alec

@girish said in SFTP Missing for some apps?:

all you have to do is open up the Web Terminal for InvoiceNinja and then edit /app/data/.env using your favorite editor (vim/nano).

Hi Girish, Thanks for your quick answer. I did get as far as trying to edit the .env file with Vim in the Cloudron terminal. I think specifically with the .env file in Invoice Ninja that it's read only in Cloudron. Let me know if I'm wrong.

In terms of SFTP access, I'd really like to be able to make some manual adjustments to the back end CSS of Invoice Ninja for usability. @nebulon suggested that I suggest some changes to Invoice Ninja on their github repository to encourage them to include the code in production. I'm not sure how Hillel Coren (lead developer) will be to do so though as he's move on from version 4 and is fully engaged in a huge rewrite for mobile in version 5 (not sure how keen users will be to do their invoicing from a mobile device on a mobile screen so it seems a strange direction).

For those reading this thread, my conversation with nebulon turned up another very interesting option. Downloading a copy of the Cloudron version of an app, making changes locally and then replacing the Cloudron code with the local version. Hopefully either Girish or Nebulon will post a link to additional documentation and a better technical description of how to do this.

Hi @girish

We have faced a serious and very frustrating issue with InvoiceNinja since we started using it a couple of years ago as our main billing system. We like InvoiceNinja but our clients simply cannot reliably receive emails from our Cloudron (despite a correct SPF record, part of the issue is the DigitalOcean IP ranges which are blacklisted with major providers). No problem we have a SendGrid account with dedicated IP and DMARC/DKIM/SPF all locked down and a good sender reputation.

What happens though is that the SMTP settings in InvoiceNinja would get set back to invoices@ourdomainname.org and Cloudron default SMTP instead of invoices@ourdomainname.com with SendGrid.

https://invoices.ourdomainname.org/settings/system_settings

Result: our clients don't get their invoices.

I thought it was InvoiceNinja updates for the first year. But we finally figure out that it's any restart of the InvoiceNinja server (and by default the server). There is an ENV file which contains the email settings.

We tried everything to edit that file and remove the email settings. No luck. Adding the email settings lower down to write them over with the correct information also does not work as that ENV file is reset on every restart.

It's the .env file that is overwritten by Cloudron. This is done in the docker using /app/pkg/start.sh where it says:

# Settings to be updated on every run.
echo "==> Update env file for database and email configs"
sed -e "s|.*\(APP_URL\).*|\1=${CLOUDRON_APP_ORIGIN}|g" \
    -e "s|.*\(DB_TYPE\).*|\1=mysql|g" \
    -e "s|.*\(DB_HOST\).*|\1=${CLOUDRON_MYSQL_HOST}:${CLOUDRON_MYSQL_PORT}|g" \
    -e "s|.*\(DB_DATABASE\).*|\1=${CLOUDRON_MYSQL_DATABASE}|g" \
    -e "s|.*\(DB_USERNAME\).*|\1=${CLOUDRON_MYSQL_USERNAME}|g" \
    -e "s|.*\(DB_PASSWORD\).*|\1=${CLOUDRON_MYSQL_PASSWORD}|g" \
    -e "s|.*\(MAIL_DRIVER\).*|\1=smtp|g" \
    -e "s|.*\(MAIL_PORT\).*|\1=${CLOUDRON_MAIL_SMTP_PORT}|g" \
    -e "s|.*\(MAIL_ENCRYPTION\).*|\1=|g" \
    -e "s|.*\(MAIL_HOST\).*|\1=${CLOUDRON_MAIL_SMTP_SERVER}|g" \
    -e "s|.*\(MAIL_USERNAME\).*|\1=${CLOUDRON_MAIL_SMTP_USERNAME}|g" \
    -e "s|.*\(MAIL_FROM_ADDRESS\).*|\1=${CLOUDRON_MAIL_FROM}|g" \
    -e "s|.*\(MAIL_PASSWORD\).*|\1=${CLOUDRON_MAIL_SMTP_PASSWORD}|g" \
    -e "s|.*\(REQUIRE_HTTPS\).*|\1=true|g" \
    -i /app/data/env

This is madness.

What we finally had to do was write a cronjob to set the email settings back to our real SendGrid credentials 30 seconds after restart.

# ll /app/pkg/start.sh
-rwxrwxr-x 1 root root 4552 Jan 19  2023 /app/pkg/start.sh*

So we used the Cloudron Cron feature to change the /app/data/env file periodically:

* * * * * 
sed -e "s|.*\(MAIL_DRIVER\).*|\1=smtp|g" -e "s|.*\(MAIL_PORT\).*|\1=465|g" -e "s|.*\(MAIL_ENCRYPTION\).*|\1=ssl|g" -e "s|.*\(MAIL_HOST\).*|\1=smtp.sendgrid.net|g" -e "s|.*\(MAIL_USERNAME\).*|\1=apikey|g" -e "s|.*\(MAIL_FROM_ADDRESS\).*|\1=invoices@ourdomainname.com|g" -e "s|.*\(MAIL_PASSWORD\).*|\1=SG.22_Y...es|g" -i /app/data/env

Issues here

1. This behaviour is not documented as far as I can see (please point out where such documentation exists if it does exist).
2. This behaviour is admin/user unfriendly. There is a settings section for email. Fill it on first run but don't overwrite hand-configured settings!

Two of us spent a billable hour each trying to figure this out in the middle of a day. I've faced stress over this email issue half a dozen times, spending about half an hour trying to figure it out each time myself.

Cloudron is so amazing but every once in a while there is a lacuna so awful, it makes one start to question open source which is the opposite of your and our intention. This is one of those cases.

Please, please remove the email defaults from the ENV file. We've figure it out but other users won't.

I'd suggest you look into this issue for every single app which would be likely to send externally directed email (internal emails will mostly remain deliverable using the built-in server SMTP).

Thanks for all your hard work making open source (mostly) accessible in a production environment!

Alec

PS. This post is intended to be part of the solution in that it should be findable on the right keywords and until the env file is neutralised other Cloudron users can use our slightly clumsy but effective cronjob patch to make sure custom outbound email settings remain correctly configured.

That's very good news, J. Thanks.

Hi Girish,

We've run into issues with reclaimable space as well. Here's where we stand now.

# docker system df
TYPE            TOTAL     ACTIVE    SIZE      RECLAIMABLE
Images          18        18        15.99GB   4.727GB (29%)
Containers      19        19        95.79kB   0B (0%)
Local Volumes   62        58        623.4MB   3.582kB (0%)
Build Cache     0         0         0B        0B

This is after running docker image prune -a which removed 19 Images (saved 4 GB), 3 Containers, 324 Local Volumes (saved 2.2 GB). We have also already ran We tried docker volume prune and docker system prune too. These only saved about 100 MB each.

How do we reclaim the last 4.727GB of reclaimable space?

Thanks in advance for your good advice, Alec

PS. I've posted on this older thread as it's also about reclaimable space, to avoid too many duplicate threads on the forum.

@nebulon Yes, simplification (I'm a fan) is always in conflict with customisation. But come on, the email preferences in InvoiceNinja are so clear and inviting. Billing is exactly the case where one might have special SMTP. Either lock these settings in InvoiceNinja (and show an error message when trying to save with information on where to change the information) or stop overwriting these preferences.

My vote in this case would be to stop overwriting the InvoiceNinja email preferences.

You have literally made my life much, much worse for two years with these decisions. There's been unnecessary friction with our most important clients due to missing invoices and deliverability issues. Cloudron became as much a source of frustration as joy due to this one issue.

Cloudron is not just a toybox at this point. People depend on it for work. Overlooking core issues like overwriting email preferences is no longer the right thing to do if it ever was. Cloudron is not proof-of-concept, it's production software.

Thanks very much for explaining how Cloudron app email works, girish. It would be great to have this explanation with a bit more detail in the Cloudron documentation. Tracking all this down took a fair amount of time, and I had to interrupt you.

First one has to delete the files, then one has to make NextCloud delete the files (NextCloud only hides files), then one has to clean the database.

Here's our own notes:

We wanted to check what's taking so much space on the server and found the trashed items in NextCloud never seem to disappear. There does not seem to be any setting of NextCloud to do that. So you found a workaround:

First I removed content of /home/yellowtent/appsdata/230be9ee-72d6-4c54-8218-b08f8d217666/data/admin/files_trashbin on server
Then I run the command on NextCloud console on Cloudron to re-check files for the "admin" user:

sudo -u www-data php occ files:scan admin

@robi said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

@foliovision is there a setting for nextcloud to not save trash data? Or only keep it for a short time?

Yes, but it looks like Cloudron defaults to keeping data forever, requiring those of us running NextCloud to dig very deep to figure out how to get rid of deleted files. Intelligent defaults here would be useful. NextCloud is such a bear that I don't blame Cloudron admins for throwing up their arms and telling users, "You want to run NextCloud? Learn how to admin it properly." InvoiceNinja is in another category. It's relatively easy to setup and easy to use.

Yes, users should not be changing database settings, that's too tied into the automation as @girish suggested. Totally understand. Email configuration can be set to default on first launch but then should not be tinkered with if the user changes email settings, particularly for apps where email is core to their functionality.

@nebulon already recognised this issue, which is why there are custom settings for email programs. nebulon should realise that email deliverability has changed in a cardinal way since he set up Cloudron. Email has gone from most email gets through unless there was a previous major spam issue from a given account to email does not get through unless the major ISP has been paid off (partnership relationships) or there is a full SPF/DKIM/DMARC setup, which is not entirely possible with Cloudron from what I can remember (we have setup what we can) AND you are not hosted on DigitalOcean (the recommended provider of Cloudron).

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

In an ideal world, we would probably hide that UI to avoid all this, but we are not the developers of those apps and contributing upstream

This is a backwards approach. It would be completely in Cloudron's control to check the .env variables for every app. If those variables change, Cloudron should:

1. notify the admin user with a screen on first startup alerting the user that key variables have changed (which ones)
2. send an email to the admin user alerting the user that key variables have been changed

I don't think local admins should have control over database settings but we should certainly be alerted if they are being changed back and forth.

It would also be very simple to allow users to enable or disable centralised email rewritten in each app. It's just one checkmark in the admin interface for the particular install.

That you have not spent the time to think about these issues and/or fix them is deeply disappointing. Your insouciance about how awful the situation is now is a deep indictment of the open source model (and this is from someone who has spent most of his adult life both creating open source and funding it).

Cloudron is sinking under too many apps and not enough care and housekeeping around the key ones you do support. I know the too-much-to-support issue first-hand. Due to WordPress making it ever more painful to keep our FOSS plugins both up to date and active in the directory (there were code change barriers, but even more administrative barriers), we had to retire about ten of them. Users can't find them or use them.

@girish does a great job running around to help us all and give us tips. It's outstanding. Systemic issues like this need to be addressed in a deeper way and not waved away with one blithe hand, or swept under the carpet.

Don't let the yes-men flatter you into complaisance. Unmerited pats-on-the-back is not how one improves a system or makes progress.

Hi Girish and fellow Rocket.Chat admins,

We have run into space issues with our 100GB Digial Ocean Cloudron droplet where we run NextCloud, Rocket.Chat and a few other apps. We cleaned out the Next.Cloud trash (thanks to the good advice on this forum) and then removed some more big documents. We're in the clear now but would like to finish the clean up.

We know that we have a lot of uploaded files in our Rocket.Chat install but we were unable to find them. The files are supposed to be flat files according to Rocket.Chat documentation. We can't find them by name, searching the server via terminal.

Could anyone tell us how we should best 1. find our Rocket.Chat uploads and 2. manage our files uploaded on the server via Rocket.chat?

Thanks for any advice in advance.

Alec

Thanks girish! It's highly ironic where Rocket.Chat to whom many of have turned to move our private chat outside of the All Seeing Eye of Sauron recommends storing one's file attachments:

What we recommend as the best option for the file upload system are Amazon S3 and Google Cloud Storage.

Those retention policies might work for us ("delete files only") if we warn all our users that files are temporary in Rocket.Chat and that everything important should be downloaded and saved locally.

@martinv