How does spam work with special mail folders?

d19dotca

It seems the Cloudron mail server automatically creates the following folders:

INBOX
Sent
Drafts
Archive
Trash
Spam

However, iOS and macOS (as well as supposedly the RFC specifications itself) seems to prefer the name "Junk" instead of "Spam", so when I connect my iOS device to my mail account it then creates a Junk folder and uses that automatically as where it deems junk mail to go to, and doesn't use the Spam folder it seems. I know this can be customized on macOS but it doesn't seem this is possible on iOS, so the commonality would be to then use the Junk folder it creates.

The above is really just a long way to get to my question... if I choose to use a folder named Junk for junk/spam mail instead of the Spam folder created by Cloudron by default, will the spam-filtering software on Cloudron still know to look at the Junk folder (if set as a system folder in say Rainloop or something) for junk to analyze and learn from, or is it configured to only look at the Spam folder created by default?

Hopefully the above makes sense, I probably made it overly confusing. lol. Sorry in advance if I did.

tl;dr = I can get around the non-standard method of default folder names, but more focused right now with how spam filtering will learn from spam messages if it's only configured to look at one folder in particular and not the Junk one created and used by iOS when marking a message as junk.''

EDIT: I have now observed that if I try to use the Junk folder as the system spam/junk folder in both macOS and iOS, and then try to delete the auto-generated Spam folder, it recreates itself immediately. This makes me assume the Spam folder is required and is the only one used by SpamAssassin for categorizing/learning from junk mail. This, however, makes it a bit difficult to use with iOS Mail app in particular as it doesn't allow customizing which folder is the junk folder, it just creates one if one doesn't exist that's already called Junk. I can work around this by "moving" to Spam instead of "marking as junk" where it moves it to the special junk folder, but this isn't ideal. Though certainly livable.

girish

@d19dotca Thanks for the details! Let us get back by testing it out on a Mac.

imc67

Hi @girish whats the status of this investigation because today I noticed the same issue?

Kind regards,

Marcel

d19dotca

Hey @girish - it's almost a year since this topic was posted. Last it left off was you running some tests. It's a minor issue but would be very helpful if we could choose our own special spam folder so that it works better with iOS and macOS (iOS in particular as on iOS you can't actually pick the spam folder). But if Cloudron won't let us choose the spam folder that the spam learning will learn from, then there's no point and we'd need to wait for that feature to be introduced. Any insight into this one?

girish

@d19dotca @imc67 I don't have a Mac or iOS to test but the IMAP configuration is just:

    mailbox Spam {
        auto = subscribe
        special_use = \Junk
    }

So, the directory is called Spam and special use flag is set per RFC 6154. Is that not enough for the Mail client?

On Thunderbird, I need to configure it like this:

d19dotca

@girish That works on macOS the same way as it does for you on Thunderbird where one can set the junk folder. Unfortunately on iOS, it insists on creating its own junk folder if it doesn’t find one when using the junk button, and so it can’t be set to use the special spam folder that was auto-generated in Cloudron. This means on iOS I have to take a few extra steps/taps to file the message away in the spam folder, because if I hit the designated junk button it will create its own folder since one does not exists that it recognizes.

This issue presents a user experience conflict.

One way I would typically work this in any other email service solution is to create the folder iOS expects and use that for junk. However if I were to do this in Cloudron, I’m worried (and no answer has been provided to confirm this yet) that the actual spam learning that’s automatically done won’t actually take place, as I suspect since the spam folder can not be deleted that it is the only folder spamassassin is set to learn from.

I think a solution here would be to allow a Cloudron user (or maybe just admins / domain admins) to create or choose the special folder that spamassassin will learn from for spam or ham. Basically, giving users more insight and control into spam services. This was also addressed in a few other threads on spam and improving its usability for users by exposing some of spamassassin features to admins or users and allowing them to set their own spam preferences.

girish

Ah, I see. I happened to get hold of a Mac today and can confirm that I can select the Spam folder and it works correctly as per https://support.apple.com/en-us/HT202325

For iOS, it seems there is no solution. For example, someone has similar problem with outlook.com addresses as well - https://discussions.apple.com/thread/7805226. And per https://clients.websavers.ca/whmcs/knowledgebase/231/Using-iOSandsharp039s-Junk-mail-folder-alongside-Spam.html there is no way to configure Junk mailbox name in iOS...

Currently, Cloudron will only learn (and unlearn) from "Spam" folder. It's easy to add Junk folder as well for training purposes but have to then figure out which folder incoming spam should end up.

girish

@d19dotca How does this work with gmail accounts on iOS btw? Or is it that they have "pre-configured" provide listing when you create an account and thus hard coded spam folder name accordingly (because gmail uses Spam and not Junk afaik).

girish

OK, here's a workaround for iOS.

SSH into server and create symlink from Junk to Spam

  # cd /home/yellowtent/boxdata/mail/vmail/{mailboxname}/mail
  # ln -s .Spam .Junk`
  # docker restart mail

You have to do above for all mailboxes. This will make both Spam and Junk folder identical and they will both appear in mail clients (unfortunate...). But atleast, it will make iOS client work.

We are looking into migrating the folder names for a future Cloudron release (there is a bug that we used "." as the separator in folder names. this makes ACLs and shared mailboxes unviable). Maybe at that time, we can look into making these folder names more customizable.

d19dotca

@girish That's an interesting workaround idea. I assume the only downside to that workaround though is that anytime the Cloudron server restarts, those changes go away and need to be re-entered? Or do they stick around?

girish

The symlinks will stick around. They are not part of the backups though. So you have to re-create the symlinks if you restore cloudron to another server.

I got the symlink idea from here https://wiki.dovecot.org/SharedMailboxes/Symlinks . I tried it on my server and it does work. Works quite nicely on thunderbird because it doesn't auto-subscribe to Spam folder by default even (so I don't have to adjust my spam settings for new accounts).

imc67

found this, is this something to configure ourselfves in Haraka config?

https://apple.stackexchange.com/questions/105145/what-are-the-default-special-folder-names-for-imap-accounts-in-mail-app-like-dr#105146

d19dotca

@imc67 That SO post is great, I had seen that many years ago when running a mail server before and couldn't find it to raise in this forum. haha. Thanks for finding that!

@girish Per the RFC mentioned in that first response to the OP in SO, it indicates that the folder really should be named "Junk", not "Spam", and I think this is partly the root cause of why the default Mail app in macOS and iOS do not see it by default - because it's following RFC and expecting the folder to be named "Junk".

girish

The folder name is different from the special_use flags. The RFC is suggesting the use of the "\junk" flag. This is what Cloudron also does. The mailbox name is not suggested in the RFC AFAICT. Can you tell me which section you are referring to?

d19dotca

@girish Ah, I believe you are correct there. I think I just typically see the names the same so I mixed them up. I still think though that it should be named "Junk" instead of "Spam" as that is what I have used in the past and it works well. Even your Thunderbird screenshot indicates they expect it to be named "Junk" too as they don't use the word "Spam". Of course that's just my interpretation and past experience, but I believe you're correct that this isn't a defined requirement in the RFC.

I really think admins should have more control over this, such as setting up their own default folder list, for example, or the ability to mark multiple folders as '\Junk' and the ability to remove the Spam folder in favour of naming it Junk.

girish

@d19dotca said in How does spam work with special mail folders?:

I really think admins should have more control over this, such as setting up their own default folder list, for example, or the ability to mark multiple folders as '\Junk' and the ability to remove the Spam folder in favour of naming it Junk.

I can get behind this. But I am quite practical about these things. There's nothing we can do to fix iOS and mac, so I am OK to even rename the folder to Junk if it increases compatibility. In the coming release (5.3), I have put some "state" information that will help us migrate to the release after. Existing installations can be on Spam and new installations can use Junk. I am doing the same for the mailbox separator. (moving things from . to / to make mailbox sharing work).

necrevistonnezr

@girish said in How does spam work with special mail folders?:

OK, here's a workaround for iOS.

SSH into server and create symlink from Junk to Spam
  # cd /home/yellowtent/boxdata/mail/vmail/{mailboxname}/mail
  # ln -s .Spam .Junk`
  # docker restart mail
You have to do above for all mailboxes. This will make both Spam and Junk folder identical and they will both appear in mail clients (unfortunate...). But atleast, it will make iOS client work.

We are looking into migrating the folder names for a future Cloudron release (there is a bug that we used "." as the separator in folder names. this makes ACLs and shared mailboxes unviable). Maybe at that time, we can look into making these folder names more customizable.

BTW there's similar "problem" for "Sent" / "Sent Messages" and "Trash" / "Deleted Messages"...

d19dotca

@necrevistonnezr True. The difference though is at least it can be worked around on iOS and macOS as you can set the folder roles. Can’t do that with spam/junk folder unfortunately.

necrevistonnezr

@girish said in How does spam work with special mail folders?:

OK, here's a workaround for iOS.

SSH into server and create symlink from Junk to Spam
  # cd /home/yellowtent/boxdata/mail/vmail/{mailboxname}/mail
  # ln -s .Spam .Junk`
  # docker restart mail

@girish Wouldn't it make sense to create hardlinks instead of symlinks for these folders so that you don't have duplicate folders? Ar am I overlooking something?

girish

@necrevistonnezr said in How does spam work with special mail folders?:

@girish Wouldn't it make sense to create hardlinks instead of symlinks for these folders so that you don't have duplicate folders? Ar am I overlooking something?

The cloudron backup logic does not understand hardlinks. It would then end up taking backup of your junk folder twice. This is unlike symlinks which is ignored by the backup code. Neither the symlink nor the contents of a symlink are backed up.

This does bring up an important point that if you move your mail server to a different server, you have to remember to create this symlink by hand again!

Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.

Cloudron Forum

How does spam work with special mail folders?