Solved What is the intention of /home/yellowtent/boxdata/mail/spamd ?

[d19dotca]

I was just browsing around the file system out of curiosity and interest, and found /home/yellowtent/boxdata/mail/spamd and didn't see any references to this in the forum or documentation so figured it'd be a good chance to learn how that part of the Cloudron server works. I assume this is where SpamAssassin does it's learning from Spam mailboxes and stores it's Bayes tokens and such?

In my case, I saw a ton of folders in there and oddly enough for many mailboxes that don't exist such as my username@differentDomain.tld and I suspect this may have to do with a long time ago when I setup a bunch of aliases in Rainloop to quickly switch to different domains hosted on my server for postmaster/webmaster, etc. I changed tactics a long time ago though and so unsure if this is just old left-over cruft. And if so, is it safe for me to remove those to clean it out a bit?

[girish]

Yes, correct. /home/yellowtent/boxdata/mail/spamd contains:

• per-user spamassassin training information. Each spam directory contains bayes_seen and bayes_tok files (bayes is the algorithm for filtering spam).
• custom spamassassin rules in custom.cf
• spam blacklist in acl.cf

You can safely delete the random mailbox directories.

[d19dotca]

@girish Perfect! Thanks so much for the speedy and detailed reply.

[d19dotca]

@girish In looking through the directory in more detail, I see what seems like a lot of duplicates, with the expected mailbox and then the same mailbox but at the primary domain of the server. Is that as designed? Seems strange, and not 100% across all of them, but many of them.

Also what would be the worst case if I emptied that directory, would it just be that it has to re-learn the spam stuff from the users spam folders?

[robi]

is it using a lot of space?
is it causing some other filtering issue?

[girish]

@d19dotca said in What is the intention of /home/yellowtent/boxdata/mail/spamd ?:

Also what would be the worst case if I emptied that directory, would it just be that it has to re-learn the spam stuff from the users spam folders?

Yes, it just loses training data, that's all.

Can you give me an example of what you mean by "the expected mailbox and then the same mailbox but at the primary domain of the server" ?

[d19dotca]

@robi Not causing any real issues to my knowledge but I like having a “clean” system where I can. And am interested in how those directories get handled under different situations (like when a mailbox is deleted) in case not all use-cases are accounted for. Because it seems to me there are a lot of old mailbox references here not to mention the seemingly duplicates for several of them too (duplicate first part but different domain where they don’t actually have a mailbox on that domain). Thus the questions.

[d19dotca]

@girish Okay perfect, I may just delete most of them then at some point to sort of “reset” them. I’ll perhaps send you a listing of the folder over email because I’d like to protect their privacy. But as an example:

Let’s say I have a mailbox on example.com, the listing in this directory will have two in some cases for my mailbox:

• dustin@example.com
• dustin@<primaryDomainOfCloudronServer>

And I’m unsure why that is. There’s also a few that were never mailboxes so I have no idea how or why they were created. I can recognize most of them but there’s one or two I have absolutely no idea what it’s referencing as it was never a mailbox nor a name I even recognize. So it’s seem quite odd. I’ll email you a listing though and won’t delete anything yet in case you want to look into it on why and how they could have been created and such.

[robi]

@d19dotca is this a very old cloudron that went through lots of upgrades?

[d19dotca]

@robi It's maybe 1.5 years old, so it's been through a decent amount but nothing crazy, no.

[girish]

@d19dotca We had a bug a year or so ago where spam detection was being done with incorrect addresses. It has the symptoms you mention. You will see random addresses, cc addresses etc there. You can safely delete them, they won't reappear. If they reappear, let us know.

[d19dotca]

@girish Oh that's good to know, sounds good then. I'll just go ahead and empty that folder (minus a couple of mailboxes) to make it easier since it'll just regenerate what's needed/used too.

[d19dotca]

@girish Just a quick question: When would the folders get re-generated by Cloudron?

For context: I moved all the files inside of that directory to /home/ubuntu/spamd-backup/ just so I had it in case I needed to restore. So the directory was then empty, and I restarted the mail service and it auto-generated the acl.cf and custom.cf files, but I don't yet see any user mailbox folders in there. Is that expected?

[d19dotca]

Okay I see a couple of mailboxes regenerated in there now so that's good. I feel less scared. haha. Still not sure when it gets generated though, I had tried for example moving one email to my own spam folder and checked but didn't see it generated. Any insight there?

[girish]

They get created when the spam filter learnt something about the address. This can happen because of user action - user marked spam or ham (moving in and out of Spam folder). There is also nightly cron job that learns from recent mail (mail of last 3 days) of each user.

[d19dotca]

@girish Is it expected to see forwarding addresses created in there? For example, I have one directory created in there for an email that doesn't reside on my server and is used in forwarding only from a mailing list in Cloudron (not from a mailbox). Seems strange that'd be in there, no?

[girish]

@d19dotca Yes, you shouldn't see forwarding addresses in spamd directory. Do you know if this address was a mailbox before? If so, can you also double check if /home/yellowtent/boxdata/mail/vmail/<forwardingaddress> exists ?

[d19dotca]

@girish I’ll check in the morning for sure, I’ll double check if it exists somehow in vmail. It shouldn’t though as it’s an @me.com address (Apple-owned) so it’s never been something I could even host at all in the first place.

Though now that I think about it, back before Cloudron allowed mailing list to external addresses, it was a mailbox on their business domain and then a forwarding rule assigned to their @me.com address. But that mailbox was removed soon after mailing lists could forward externally and is just a mailing list since the. So maybe there’s still something in there?

I’ll update tomorrow

[girish]

Oh, the vmail and spamd directories should only ever contain addresses that are hosted on Cloudron. If you see anything not belonging to Cloudron, you can delete it . If they "re-appear", please report here, because that is some bug. I thought we fixed this bug a long time ago. I quickly checked atleast 5 Cloudrons that I could get hold of and none of them have external mailboxes (anymore).

[d19dotca]

@girish I think that’s the thing though, the one I’m mentioning I see in spamd is the @me.com address, so I guess that’s a bug then eh?

[girish]

@d19dotca Yes, that looks like a bug. Can you remove it when you ssh in the next time around and let me know when it re-surfaces?

[d19dotca]

@girish I had emptied the spamd folder the other day, so if the @me.com address is coming up then I guess a bug. I’ll check the vmail too once I can tomorrow.

[d19dotca]

@girish Okay I checked in vmail today and it's all the expected mailboxes. So the @me.com address is only present in spamd. I've deleted it again from spamd and will see if it comes back again later today.
EDIT: I just looked again in the spamd folder and actually found three others in there that went to domains I've never hosted (i.e. they're external domains used in forwarding only from mailing lists and not mailboxes). Definitely seems like a bug, but I've removed all of them now and will see if any of those come back.

[d19dotca]

@girish Okay, I can confirm it's back again (just one of them but I assume if I wait longer the others will show up too). So definitely seems to be a bug. Is there anything I can do to help troubleshoot / diagnose it? Do you want SSH access again?

Here's a screenshot showing some of the mailboxes in the spamd folder, can confirm the @me.com in there which obviously I don't host that domain since Apple does.

One thing I noticed is the permissions all seem a bit inconsistent, not sure if that's expected but I'd have assumed they'd be consistent. So that may or may not be another bug too?

[d19dotca]

@girish - just wanted to follow up on this. Anything needed of me for fixing this defect?

[girish]

@d19dotca I will try to reproduce this, it's in my TODO list.

[d19dotca]

@girish Hi Girish. Just following up on this. Doesn't seem like it's been fixed yet.

[d19dotca]

@girish Just following up on this. Seems to still be an issue in Cloudron (or at least my instance of it).

[d19dotca]

(still an issue - also possibly related to https://forum.cloudron.io/topic/4921/bayes_00-rule-set-for-some-mail-to-mailing-list-no-mailbox/1 ?)