Email configured, but no notification

[umnz]

So I've nuked my test instance of OpenProject so that I could relaunch it with app configured users instead of using Cloudron User Directory and I can confirm that emails invites do not go out -- regardless. Sending a test email via the OpenProject back end works - but creating a new user from the backend and sending them an invite doesn't. Manually resending the invitation link also doesn't work (and it doesn't throw an error either).

There is nothing in the logs to indicate there is a problem.

[nebulon]

Not an OpenProject expert, but maybe there is a setting for invites or such? If the test email works I suppose the package setup is correct. Otherwise you may also reach out to OpenProject upstream to clarify what is expected to happen.

[umnz]

@nebulon Good call. I checked upstream to see if this was a common issue (I had assumed it was Cloudron related) - and here's what I found.

Docker / Worker / Port Issue
https://community.openproject.org/topics/11644

Using this test URL you can run a health check on an OpenProject instance:
https://openproject.example.com/health_checks/all

full
  database: PASSED Schema version: 20240611105232 (0.005s)
  default: PASSED Application is running (0.000s)
  mail: PASSED ActionMailer::Base check to mail:2525 successful (0.004s)
  puma: PASSED N/A as Puma is not used. (0.000s) (OPTIONAL)
  worker: FAILED No good_job processes are active. (0.048s)
  worker_backed_up: FAILED 199 jobs are waiting to be picked up for more than 5 minutes. (0.024s) (OPTIONAL)

This issue is caused by the worker not working, and this is happening on fresh deployments of OpenProject installed on Ubuntu 24.04 LTS.

Can you please check on a fresh install of Cloudron at your end to see whether or not your worker is running?

Sending a test email will work - as it is called directly and not via the worker. But creating & inviting a test user (plus many other operations) will add to the worker's queue which are failing.

Also, as others have reported, there is a recurring issue in the Cloudron logs for the worker:

Jul 17 14:35:13 => Run cron job
Jul 17 14:35:27 rake aborted!
Jul 17 14:35:27 Don't know how to build task 'jobs:workoff' (See the list of available tasks with `rake --tasks`)
Jul 17 14:35:27 (See full trace by running task with --trace)

Note the above Cron is Cloudron's call - not one I have set up. A fresh installation will get this error and I believe it is the source of the emails not sending out when queued.

[umnz]

Further reading about the cronjob here:
https://community.openproject.org/topics/5276

root@xxx:/app/code# bundle exec rake jobs:workoff
rake aborted!
Don't know how to build task 'jobs:workoff' (See the list of available tasks with `rake --tasks`)
/app/code/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/root/.rbenv/versions/3.3.2/bin/bundle:25:in `load'
/root/.rbenv/versions/3.3.2/bin/bundle:25:in `<main>'
(See full trace by running task with --trace)

[umnz]

Assuming it was a cron related issue, I found this post:
https://community.openproject.org/topics/5276?r=5306#message-5306

Where it was recommended to run the following commands - to ensure various services were running:

RAILS_ENV="production" bundle exec rake db:create:all
RAILS_ENV="production" bundle exec rake db:migrate
RAILS_ENV="production" bundle exec rake db:seed
RAILS_ENV="production" bundle exec rake assets:precompile

All but the last command is successful. I get an error which aborts the process:

root@xxx:/app/code# RAILS_ENV="production" bundle exec rake assets:precompile
I, [2024-07-17T03:15:54.881512 #446]  INFO -- : Increasing database pool size to 17 to match max threads
Exporting I18n.js locales
=> Done in 12.29s
Linking frontend plugins
Cleaning linked target directory /app/code/frontend/src/app/features/plugins/linked
Linking frontend of OpenProject plugin budgets to /app/code/frontend/src/app/features/plugins/linked/budgets.
rake aborted!
Errno::EROFS: Read-only file system @ rb_file_s_symlink - (/app/code/modules/budgets/frontend/module, /app/code/frontend/src/app/features/plugins/linked/budgets/module) (Errno::EROFS)
/app/code/lib/open_project/plugins/frontend_linking/generator.rb:74:in `block (2 levels) in regenerate_angular_links'
/app/code/lib/open_project/plugins/frontend_linking/generator.rb:69:in `each'
/app/code/lib/open_project/plugins/frontend_linking/generator.rb:69:in `block in regenerate_angular_links'
/app/code/lib/open_project/plugins/frontend_linking/generator.rb:61:in `regenerate_angular_links'
/app/code/lib/open_project/plugins/frontend_linking/generator.rb:52:in `regenerate!'
/app/code/lib/open_project/plugins/frontend_linking.rb:35:in `regenerate!'
/app/code/lib/tasks/plugins.rake:35:in `block (3 levels) in <top (required)>'
/app/code/lib/tasks/assets.rake:61:in `block (2 levels) in <top (required)>'
/app/code/vendor/bundle/ruby/3.3.0/gems/rake-13.2.1/exe/rake:27:in `<top (required)>'
/root/.rbenv/versions/3.3.2/bin/bundle:25:in `load'
/root/.rbenv/versions/3.3.2/bin/bundle:25:in `<main>'
Tasks: TOP => openproject:plugins:register_frontend
(See full trace by running task with --trace)

I'm not 100% sure what happens during the deployment of OpenProject via Cloudron so I can only leave this with you guys to help me understand how to get the cron working correctly.

[nebulon]

Thanks for all the research! Most rake commands are run already in Cloudron, however the job workers are indeed not run.

The command would be RAILS_ENV="production" bundle exec good_job start which is a longstanding task, so we need to add support for that in the package, which currently is working with one apache+passenger and a cron script.

We will also start using the better healtheck path you mentioned.

[umnz]

Thanks @nebulon
Using that command via terminal allowed me to clear out the backlog of tasks -- but it didn't help solve system email mystery. I left that worker running while attempting to send an invite and also a password reset and still no emails are leaving the system. Of course, directly calling a test email works. So I'm kind of at a loss at this point.

[umnz]

I've found some additional information here that might be of help @nebulon but I'm not sure how to run the openproject commands they mention.

https://www.openproject.org/docs/installation-and-operations/installation-faq/#i-dont-receive-emails-test-email-works-fine-but-not-the-one-for-work-package-updates-what-can-i-do

[nebulon]

I am currently reworking the package to be closer to upstream. Using puma instead of passenger for example.
For the imap fetching in cron, I will see how we can integrate https://github.com/opf/openproject/blob/3757cf397f0b7d048aa7b180cfb100c06103eb3c/docker/prod/cron

[nebulon]

The email sending via the task worker should be fixed now with latest package release.

[umnz]

Hi @nebulon - thank you! Just confirming after updating, the emails are now being sent.