Discourse and incoming email

jenny_molloy

Hi

I need incoming email to function on Discourse and it seems like there are two options:

set up a POP3 inbox, which I can't do on Cloudron
add this module for direct delivery of email, which I also can't do on Cloudron

At the moment I will have setup an external POP3 service and hope for the best. Would you consider including the direct email delivery module in future Discourse setups?

Any tips in the meantime from people who have got this to work?
Many thanks!

Jenny

jimcavoli

From a cursory glance, it looks like the latter may be possible, though with some custom records and the like, it would be a somewhat more involved setup. Frankly, the most appropriate way to do that would be for box to have a DNS add-on that could handle things like this...also would help automate install on certain things requiring SRV or TXT records and the like. After that, there's the problem of handling the transport on port 25 which would require another round of changes to the shared mail server (haraka) to answer correctly and pass those emails off to the app...also not impossible, but another thing to do. Turns into a bit of a feat, but it is possible. I see no reason it couldn't be pulled off in a generally-useful way, but it would definitely be a serious set of work and not be done super fast. The ROI for other use cases than just discourse is likely to be a major consideration along the way. The mail server changes are more the issue than the DNS ones, imo. Worth thinking about though.

girish

I cannot think of a good solution immediately. But why exactly does discourse require POP3 over IMAP? Seems so strange, even really old things support IMAP these days, just very surprised.

jdaviescoates

@girish said in Discourse and incoming email:

I cannot think of a good solution immediately. But why exactly does discourse require POP3 over IMAP? Seems so strange, even really old things support IMAP these days, just very surprised.

Good question. And yes, it does seem odd, doesn't it!

jimcavoli

@girish There is pending IMAP support in https://github.com/discourse/discourse/pull/8301 which has been going through some back-and forth for review since November, but it's looking pretty positive as far as actual merge/release possibility in the next few months. If that happens, it's definitely the easiest option on the board.

jenny_molloy

Thanks for the advice! I managed to get inbound mail coming in now by setting up a subdomain and external POP mail box, which fixes an immediate problem regarding how we were planning to use the forum with specific category emails.

The main outstanding issue now with allowing incoming replies is that Cloudron outbound SMTP won't accept a "reply to" address that is a subdomain with plus addresses/VERPs (I guess both of these are separate issues).

I am trying Amazon SES as the SMTP server for the whole domain and will see if that solves the issue and report back.

Jenny

girish

@jenny_molloy For subdomain emails, simply add the subdomain itself as a domain in Cloudron. Then the mail server will started allowing that subdomain for outbound. You can even setup SES relay in mail -> domain -> SES.

jenny_molloy

Great - adding the subdomain in Cloudron and setting up Amazon SES login for that fixed the issue so I can now both send (via SES) and receive emails (via a Zoho pop mail box), including reply by email mode.
Thanks for your advice everyone!

bmann

I am in the same boat.

Discourse has IMAP support but only for group inboxes, not for "reply by email".

Discourse was one of the big apps I want to be running on Cloudron, and having to add an external email provider just for POP3 really sucks.

Discourse has their "mail receiver" docker container that runs all of this for you if you self install elsewhere, and works really well.

"Don't run Discourse on Cloudron" seems like a bad solution. Is this on a backlog somewhere to solve?

jdaviescoates

@bmann I hadn't realised "reply by email" doesn't work on Cloudron, but if that's the case then that is a rather major issue that really ought to have @staff attention imho. It's a core feature of Discourse that many people familiar with Discourse have come to expect.

I'm not sure, but I think it's basically the same/ a related issue @jimcavoli has been hitting when trying to package Loomio, see:

https://forum.cloudron.io/post/35429

robi

Does "reply-by-email" work for NodeBB?

girish

@bmann I guess https://github.com/discourse/mail-receiver is the repo? Did I understand correctly that it can be run on a separate server and hooked up to Cloudron+Discourse install? (Of course, ideally, we can have this all part of the Cloudron package itself or maybe a separate Cloudron package)

eddowding

Argh. Just gone through a lengthy process of getting Discourse set up - and it was a big part of choosing Cloudron - only to find this thread.

This is challenging, even though there is an answer.

Any solutions?
Does NodeBB do inbound? On Cloudron?
Can this be done with Mailgun instead?

girish

@eddowding nodebb doesn't do inbound afaik. Incidentally, I was looking into making our recvmail addon work. Let me look into if it's easy to enable POP3 at a mailbox level.

eddowding

@girish thanks - here's hoping.

Would this be a quick-to-deploy solution?

jdaviescoates

@eddowding said in Discourse and incoming email:

Can this be done with Mailgun instead?

I've not tried it yet, but I think there is a new option when installing apps to not use Cloudron for mail at all and to just configure it all within the app itself. I guess that might work @staff ?

girish

@jdaviescoates That feature is for used for configuring apps that send email. @eddowding wants a way for an app to receive email.

girish

@eddowding yes, it will be part of update (if we implement it). Just to double check, this feature only makes sense if your domain's email is also hosted on Cloudron. Is that the case with your setup?

jdaviescoates

@girish said in Discourse and incoming email:

@jdaviescoates That feature is for used for configuring apps that send email. @eddowding wants a way for an app to receive email.

I realised @eddowding wants to receive email (and I think it's really quite bad that Discourse is a Stable App but doesn't have some of it's basic features packaged, nor any warning of those missing features. Indeed when installing that app it says "Use it as a: mailing list" which you can't do if it can't receive mail), I just figured Mailgun details or whatever could be entered somewhere into Discourse settings and that might work. I guess you're saying that wouldn't work but I don't think I really understand why that would be the case

girish

@jdaviescoates It's debatable if mailing list is a basic feature for forum software (though it seems the description comes from https://github.com/discourse/discourse#readme and I am reluctant to remove it since this is what the upstream devs think )

That said, you can use it as a mailing list, just not if the email domain is hosted on Cloudron since we don't support POP3 yet. I also don't know why they only support POP3 when all other apps support IMAP (gitlab, freescout, espocrm for example). Anyone have any background on this?

eddowding

@girish Yep email is / can be hosted here too for this domain.

eddowding

@girish https://meta.discourse.org/t/configuring-reply-via-email/42026

jdaviescoates

@girish said in Discourse and incoming email:

I also don't know why they only support POP3

Yes, it does seem odd.

Although it sounds like it's not the only way:

Pushing emails via the Discourse API

Alternatively, you can push emails to Discourse via the API. For that, you will need
To check the manual_polling_enabled site setting.
A master API key 138 or an API key associated to a staff member.
Then, to push an email to Discourse, you need to pass the raw email content as the “email” parameter when POSTing to “/admin/email/handle_mail”. The endpoint accepts application/json (recommended), multipart/form-data, or application/x-www-urlencoded request bodies (with an appropriate request Content-Type header, naturally).

You’re done

From the link @eddowding posted above

Edit: also, this thread should not be marked as solved, as this issue has not yet been solved.

eddowding

I'd love to use the API but it's far from obvious if that's actually possible on Cloudron. @jenny_molloy's opening post suggests it's not.

https://meta.discourse.org/t/direct-delivery-incoming-email-for-self-hosted-sites/49487

@jdaviescoates

girish

POP3 is enabled in the next release. Though, I haven't tried if Discourse works well with that.

nebulon

To give an update on this. Currently the package lacks the pre-setup of the pop3 configs, this is under way, however even then I wasn't able to make use of that feature properly, despite the app receiving the emails. May just be that I haven't fully grasped how it is supposed to work.

jdaviescoates

@nebulon great that this is finally moving forward. IMHO it's really very important that we get this working as it's a core feature of Discourse.

Let me know if I can help testing

nebulon

For a start I have published a new app package which does the pre-setup of the email receiving part.

Just to be clear, the transactional emails for discourse are using a different email address.

So far I haven't been able to configure it to actually accept any emails for forum threads, however it does process it now, so maybe I am just missing something small.

ntnsndr

@jdaviescoates Me too!

mononym

Hello. I was thinking of setting up a Discourse forum especially because it can also be used as a mailing list – i.e. a reply by a user via email corresponds to a forum reply if I understood well. I don’t mind using a third party email account or the cloudron one.
But by reading this thread I can’t understand if the email integration for Discourse on Cloudron actually works. Is it only a specific or a general issue? I would be delighted if somebody could illuminate me about this. Thx!

mononym

@mononym to myself
It should work it seems : https://forum.cloudron.io/post/42987

mononym

Hello again @jdaviescoates & @nebulon.
Thanks for all the efforts for making this work !

I try to get my head around the functionality to reply by email and I think I'm missing something out there.

The POP3 settings are enabled in the Discourse instance. They correspond to the mailbox in the app's configuration panel (hosted on cloudron).

The mailinglist function works as far as it concerns sending an email to each user for each new topic creation. The second functionality would be to be able to reply to this topic by email. In order to do so, Discourse asks to activate the setting reply by email enabled in Settings/Email.

The following is where problems occur :
This setting only works if a reply by email address is defined. I used the provided template from Discourse %{reply_key}@reply.example.com with my domain. Each new topic corresponds to a reply_key which is the address a user has to send his answer to in order to comment on a topic.

Sending an email reply to let's say ui9cflk7c0obwgvouw0o@reply.example.com doesn't work. The mail cannot be delivered :
550 5.0.350 Remote server returned an error -> 550 No such address ...and how could it with an non-existing email address.

Is there any setting I'm missing in Discourse to get this work with a permanent email address ?

Update:
Following this thread, I added an alias to my cloudron mail address (and activated POP3). Besides forum@example.com I added the alias replies@example.com. In Discourse I add this email in Settings/Email pop3 polling username.

These settings break even the sending when new posts are made: 550 Authenticated user forum@example.com cannot send mail as replies+verp-9ab2aa4b4eae14ee9580a918df2cead7@example.com

robi

@mononym for that you need to turn on masquerading in the email config for that domain so forum@ can send email as replies@

eddowding

I'm finding Cloudron increasingly frustrating since it's unable to do many of the things that I want it to do.

There are 6 ways to host code and 2 ways to monitor uptime, but not much for communities.

Is it so hard to make Discourse work as it's supposed to?
Or to have a way to easily spin up docker images?

This issue has been open for two years. Suggestions that people could offer bounties have been rejected; going open source has been rejected.. should I just give up on there being a solution to this?

girish

@eddowding are you referring to installing/running discourse? Or incoming email configuration ? The former should work just fine. The latter was already implemented a while ago - https://docs.cloudron.io/apps/discourse/#incoming-email-setup . I am not sure why this thread remains unsolved but as a heads up, using a forum software as an email list ... has it's issues. From what I remember of testing this feature, discourse has lots of strange and edge case behavior when processing emails (like if you remove email body when replying, it may not do what you expect etc).

Would help to have more details about what problem(s) you are facing.

eddowding

@girish Ah in that case I take it back! (Well, some of it .. I still have concerns re software mix, but ... )

I've now got inbound email setup, but it doesn't seem allow catch-alls (even though the setting is set on cloudron) so each category email needs to be set up as an alias.

I also had to enable masquerading in cloudron email settings, which is probably worth mentioning in the setup docs.

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Solved Discourse and incoming email