Automated env configuration destroys InvoiceNinja custom mail configuration on every restart

[foliovision]

Hi @girish

We have faced a serious and very frustrating issue with InvoiceNinja since we started using it a couple of years ago as our main billing system. We like InvoiceNinja but our clients simply cannot reliably receive emails from our Cloudron (despite a correct SPF record, part of the issue is the DigitalOcean IP ranges which are blacklisted with major providers). No problem we have a SendGrid account with dedicated IP and DMARC/DKIM/SPF all locked down and a good sender reputation.

What happens though is that the SMTP settings in InvoiceNinja would get set back to invoices@ourdomainname.org and Cloudron default SMTP instead of invoices@ourdomainname.com with SendGrid.

https://invoices.ourdomainname.org/settings/system_settings

Result: our clients don't get their invoices.

I thought it was InvoiceNinja updates for the first year. But we finally figure out that it's any restart of the InvoiceNinja server (and by default the server). There is an ENV file which contains the email settings.

We tried everything to edit that file and remove the email settings. No luck. Adding the email settings lower down to write them over with the correct information also does not work as that ENV file is reset on every restart.

It's the .env file that is overwritten by Cloudron. This is done in the docker using /app/pkg/start.sh where it says:

# Settings to be updated on every run.
echo "==> Update env file for database and email configs"
sed -e "s|.*\(APP_URL\).*|\1=${CLOUDRON_APP_ORIGIN}|g" \
    -e "s|.*\(DB_TYPE\).*|\1=mysql|g" \
    -e "s|.*\(DB_HOST\).*|\1=${CLOUDRON_MYSQL_HOST}:${CLOUDRON_MYSQL_PORT}|g" \
    -e "s|.*\(DB_DATABASE\).*|\1=${CLOUDRON_MYSQL_DATABASE}|g" \
    -e "s|.*\(DB_USERNAME\).*|\1=${CLOUDRON_MYSQL_USERNAME}|g" \
    -e "s|.*\(DB_PASSWORD\).*|\1=${CLOUDRON_MYSQL_PASSWORD}|g" \
    -e "s|.*\(MAIL_DRIVER\).*|\1=smtp|g" \
    -e "s|.*\(MAIL_PORT\).*|\1=${CLOUDRON_MAIL_SMTP_PORT}|g" \
    -e "s|.*\(MAIL_ENCRYPTION\).*|\1=|g" \
    -e "s|.*\(MAIL_HOST\).*|\1=${CLOUDRON_MAIL_SMTP_SERVER}|g" \
    -e "s|.*\(MAIL_USERNAME\).*|\1=${CLOUDRON_MAIL_SMTP_USERNAME}|g" \
    -e "s|.*\(MAIL_FROM_ADDRESS\).*|\1=${CLOUDRON_MAIL_FROM}|g" \
    -e "s|.*\(MAIL_PASSWORD\).*|\1=${CLOUDRON_MAIL_SMTP_PASSWORD}|g" \
    -e "s|.*\(REQUIRE_HTTPS\).*|\1=true|g" \
    -i /app/data/env

This is madness.

What we finally had to do was write a cronjob to set the email settings back to our real SendGrid credentials 30 seconds after restart.

# ll /app/pkg/start.sh
-rwxrwxr-x 1 root root 4552 Jan 19  2023 /app/pkg/start.sh*

So we used the Cloudron Cron feature to change the /app/data/env file periodically:

* * * * * 
sed -e "s|.*\(MAIL_DRIVER\).*|\1=smtp|g" -e "s|.*\(MAIL_PORT\).*|\1=465|g" -e "s|.*\(MAIL_ENCRYPTION\).*|\1=ssl|g" -e "s|.*\(MAIL_HOST\).*|\1=smtp.sendgrid.net|g" -e "s|.*\(MAIL_USERNAME\).*|\1=apikey|g" -e "s|.*\(MAIL_FROM_ADDRESS\).*|\1=invoices@ourdomainname.com|g" -e "s|.*\(MAIL_PASSWORD\).*|\1=SG.22_Y...es|g" -i /app/data/env

Issues here

1. This behaviour is not documented as far as I can see (please point out where such documentation exists if it does exist).
2. This behaviour is admin/user unfriendly. There is a settings section for email. Fill it on first run but don't overwrite hand-configured settings!

Two of us spent a billable hour each trying to figure this out in the middle of a day. I've faced stress over this email issue half a dozen times, spending about half an hour trying to figure it out each time myself.

Cloudron is so amazing but every once in a while there is a lacuna so awful, it makes one start to question open source which is the opposite of your and our intention. This is one of those cases.

Please, please remove the email defaults from the ENV file. We've figure it out but other users won't.

I'd suggest you look into this issue for every single app which would be likely to send externally directed email (internal emails will mostly remain deliverable using the built-in server SMTP).

Thanks for all your hard work making open source (mostly) accessible in a production environment!

Alec

PS. This post is intended to be part of the solution in that it should be findable on the right keywords and until the env file is neutralised other Cloudron users can use our slightly clumsy but effective cronjob patch to make sure custom outbound email settings remain correctly configured.

[luckow]

good catch. I was able to reproduce this behavior. I have tried Mail and Sendmail as an alternative to SMTP, neither of which worked for me. The only option was SMTP and my individual settings were overwritten by the data from the .env file after restarting the app. Bad behavior.

[girish]

@foliovision I think there is some confusion as to how Cloudron works. Cloudron packages, whenever possible, will automatically configure both user authentication and email configuration. The idea is to not repeat the same setup over and over again in each app.

For email, you have to configure Sendgrid in Email section of Cloudron. See https://docs.cloudron.io/email/#relay-outbound-mails . Once you do that, all apps will send email via Sendgrid. For some apps, like those which send bulk email, we have added a provision in the packaging system to say "Do not configure email for this app" . For example, mautic, mailtrain to new a few. But those aside, all apps use the outbound relay of Cloudron mail server.

In short, you don't need all the cron and other changes. Just have to configure Sendgrid in Email section of Cloudron.

[foliovision]

Hi @girish

Thanks for the additional information. This doesn't really suit us as we are fine using the Cloudron SMTP server for most apps. It's just the externally pointed ones which require SendGrid. We should set up a SendGrid user for every outbound address so it's a real pain in the neck to have this automatic configuration.

It looks to me like Cloudron is on the wrong path here. If there is email configuration for an app, it should not be zeroed like this. This turns into whack-a-mole.

At least allow us to adjust our ENV files, where we could put our own information lower down in the pecking order so that when the core values are overwritten higher by your automated routines, our values will remain.

Thanks, Alec

[girish]

We should set up a SendGrid user for every outbound address so it's a real pain in the neck to have this automatic configuration.

right, Cloudron only supports domain validated SMTP relays.

It looks to me like Cloudron is on the wrong path here. If there is email configuration for an app, it should not be zeroed like this. This turns into whack-a-mole.

Have to think/check more about how common this is. Cloudron has always configured email/database/auth in all it's packages since inception... so almost a decade by now. The values get overridden for consistency and ease of support. People tinker with values and complain things broke. Restarting the app fixes things because the packages "re-fixes" all the values. The main rationale though is that nobody wants to setup email configuration per app.

[jdaviescoates]

@girish said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

For some apps, like those which send bulk email, we have added a provision in the packaging system to say "Do not configure email for this app"

It seems to me the solution to this issue would be to just make that an option that is available on all apps?

Then @foliovision could just continue to use Cloudron email for most things but turn it off for InvoiceNinja (like you already can on some other apps).

[girish]

@jdaviescoates the option for optional email is added package by package. It's not possible to add it wholesale because it needs changes to each package. But I can add it to invoiceninja.

[jdaviescoates]

@girish said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

@jdaviescoates the option for optional email is added package by package. It's not possible to add it wholesale because it needs changes to each package.

Perhaps just add it package by package then, as part of the the update cycle? Or not, if it's too much work. Possibly better to so as you currently do and just do it as and when the need arises.

[nebulon]

One consideration here is always how much options we support for how many apps. Each such tweak setting causes support tickets and ongoing testing, which takes time away from other things. Its easy to add initially but the long tail will have an impact.

[jdaviescoates]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

Its easy to add initially but the long tail will have an impact.

Totally your call, but it seems to me that the impact could actually be less support tickets from all the people who try to change email settings within apps (this seems quite a common thing for people to do, resulting in a fair few threads on here about it) and then find out that doesn't work and get stuck because the app in question doesn't have that option yet

[foliovision]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

One consideration here is always how much options we support for how many apps. Each such tweak setting causes support tickets and ongoing testing, which takes time away from other things. Its easy to add initially but the long tail will have an impact.

Another consideration is the living hell we've been through with our InvoiceNinja instance. This is production financial software and Cloudron is willy-nilly overwriting settings.

Generally the Cloudron default email setup is unsatisfactory.

1. Cloudron recommends DigitalOcean as the default server provider.
2. Maintaining deliverability to large commercial ISP on DigitalOcean IP's is a nightmare/impossible (this is what we tried first, SendGrid only came in later).
3. Automated setup with SendGrid is not possible.

This is just tear-your-hair-out-in-frustration open-source-sucks territory.

Cloudron mysteriously overwriting our email settings has cost us about ten billable hours, disrupted client payments and communication and created a huge amount of stress for eighteen months.

Overwriting mail settings on every reset is about the craziest thing I've ever heard of. If you want to be this aggressive remove the email settings within the applications or lock them. Or warn us on every restart:

Your email settings have been overwritten. Please be sure you are happy with the current settings

and then show the current settings to us.

This libertarian attitude of it's up to you to know all your tools inside out all of the time, rather than a protect the user attitude, is one of the principal downfalls of open source (along with "make every option visible, fulfill every feature request however obscure").

There was a guy called Steve Jobs. He pioneered a method of software development which is to make applications and processes user-friendly. I call this way of designing software "intelligent defaults". Wiping email settings on restart is not an intelligent default.

Cloudron does so much of the plumbing right but this careless attitude towards email is extremely user-unfriendly, neckbeard kind of attitude, which does not fit the rest of what you get right.

Email must be just right for deliverability in 2023 (in fact any time after about 2018). Without perfect email pipelines, the production use of any communication application becomes a game of Russian roulette.

Right now there are certainly thousands of other Cloudron users going through frustration with email deliverability and email setup right now. Like us (and we are technically competent), they just don't know how they landed in the email hell they are in.

[nebulon]

Sorry if this particular issue caused you so much trouble. We are a very small team maintaining a large platform and a lot of apps with each their own special peculiarities. This is a huge field where we have to strike some balance between the vast use-cases amongst our user base. I doubt we can be compared to a company like Apple and that also not for the price point we have.

I am still not quite sure why you can't configure a Cloudron global email relay which will ensure deliverability across the whole system.

[foliovision]

@nebulon said in Automated env configuration destroys InvoiceNinja custom mail configuration on every restart:

I am still not quite sure why you can't configure a Cloudron global email relay which will ensure deliverability across the whole system.

1. Yes, we can. But it's a bit finicky with SendGrid. Moreover we only want InvoiceNinja on SendGrid at this point.
2. You still shouldn't be erasing people's email settings. This is just wrong. Particularly without actively notifying the user.

At the least, any app where you erase the email settings on restart should come up with a big notification on restart right across the screen that email settings have been zeroed with a hint of what to do about it. This would not be difficult to do across all applications as it's easy enough to run a check if email settings match env or not before overwriting them.

In terms of the platform and what you maintain, I agree it's a lot. It might be better if there were fewer applications with more robust support. Originally I spun up a Cloudron instance just to demo some open source software for internal use. We now use Cloudron in production. In production, issues like this are considerable friction and let down the user experience.

[necrevistonnezr]

Indeed, the documentation at https://docs.cloudron.io/apps/invoiceninja/ suggests you can edit the env file to your liking… At the very least, it should say what settings are not persistent

[girish]

@necrevistonnezr Customization within limits

I guess we expect users to know to not change authentication, database and email setup in apps directly. These 3 things are integrated into Cloudron since we identified these as 3 painful things to configure again and again when selfhosting apps. Database is very tied to backups, so this is close to impossible to change. Authentication is changeable only at install time. We tried to make this dynamic but couldn't since each app is special when it comes to migrating users from one auth provider to another.

Email can may be changable, still thinking about this after this thread. The reason why email is re-configured each time is because if you change domains of an app etc, the email is automatically re-configured on package start. A while ago we implemented the "Leave email configuration to the app" switch. We are open to add this switch to more and more apps as the use cases arise.

The original "expectation" that users need to know this has been a problem from day 1 We even put a splash screen immediately after Cloudron setup, but I think nobody reads those (can't blame them). Suggestions welcome...

[foliovision]

@nebulon Yes, simplification (I'm a fan) is always in conflict with customisation. But come on, the email preferences in InvoiceNinja are so clear and inviting. Billing is exactly the case where one might have special SMTP. Either lock these settings in InvoiceNinja (and show an error message when trying to save with information on where to change the information) or stop overwriting these preferences.

My vote in this case would be to stop overwriting the InvoiceNinja email preferences.

You have literally made my life much, much worse for two years with these decisions. There's been unnecessary friction with our most important clients due to missing invoices and deliverability issues. Cloudron became as much a source of frustration as joy due to this one issue.

Cloudron is not just a toybox at this point. People depend on it for work. Overlooking core issues like overwriting email preferences is no longer the right thing to do if it ever was. Cloudron is not proof-of-concept, it's production software.

[foliovision]

To follow up, some applications are just a pain-in-the-neck and are difficult to make work. For instance managing file space on NextCloud. NextCloud never wants to release files and delete them and it's very easy to run out of space and very hard to clear free space. These are complex issues and there are no simple solutions (apart from not running NextCloud which looks like it's probably our next step as the amount of time which goes into admin of NextCloud makes it a losing proposition). @girish was incredibly helpful and at least got us back to the starting line.

But in the case of InvoiceNinja, the application works, the preferences work. What was breaking InvoiceNinja was the Cloudron setup, not InvoiceNinja. Cloudron is effectively sabotaging effective work with an important app which does have its act together.

[robi]

@foliovision What was the procedure to properly clean NextCloud files? That might be useful to many here that use it.

[foliovision]

First one has to delete the files, then one has to make NextCloud delete the files (NextCloud only hides files), then one has to clean the database.

Here's our own notes:

We wanted to check what's taking so much space on the server and found the trashed items in NextCloud never seem to disappear. There does not seem to be any setting of NextCloud to do that. So you found a workaround:

First I removed content of /home/yellowtent/appsdata/230be9ee-72d6-4c54-8218-b08f8d217666/data/admin/files_trashbin on server
Then I run the command on NextCloud console on Cloudron to re-check files for the "admin" user:

sudo -u www-data php occ files:scan admin

[robi]

@foliovision is there a setting for nextcloud to not save trash data? Or only keep it for a short time?