Mailpiler - self hosted email archive

[vladimir.d]

@timconsidine I've updated Dockerfile to fix all building issues you faced with. After that everything went just fine on a fresh cloned copy.

@timconsidine said in Mailpiler - self hosted email archive:

Blundering around in the dark, but I notice that start.sh has on line 53

local SSL_CERT_DATA="/C=US/ST=Denial/L=Springfield/O=Dis/CN=www.example.com"

Should this not reference the Cloudron variable for location ?

When tls add-on is enabled in the manifest, the script doesn't generate a self signed certificate and uses the Cloudron certificate of the primary domain. I've amended it to use $CLOUDRON_APP_DOMAIN variable.

@timconsidine said in Mailpiler - self hosted email archive:

Successfully built and pushed to repository (docker in my case).
App installed on Cloudron w/o hassle .... but hangs in 'starting' mode.
So I uninstalled and tried cloudron install --image <repo>/<image>:<tag> without the --no-wait option.
This installs but similarly gets stuck

 => Wait for health check ............................^C

Out of my depth now. What to check / amend ?

It doesn't get stuck for me - I've tried to build&install the app with the recent changes.

I guess there is an issue with piler or nginx directories in /etc/, probably a wrong path or an invalid symlink or so.
You need to look at the app logs to see the issue.

[timconsidine]

@vladimir-d thank you !
I will give it another go in the morning.

[vladimir.d]

@girish It needs to expose 25 port for piler-smtp daemon. it says 25 (and other smtp ports) are reserved.
For now I've added tcpPorts field to Cloudronmanifest.json and exposed it as 12525 but I guess it won't properly work in most cases.
Is there a workaround to expose 25 port for the app?

[girish]

@vladimir-d Oh, interesting. Don't think an app can ever expose port 25 since the mail server requires port 25. Do you know why it requires port 25? Maybe I don't understand how the archiver works. I though it's pulling in all email via IMAP? Or is it that we should redirect all mail to mailpiler?

[luckow]

@girish According to my understanding of a mail archive solution, every incoming and outgoing mail must be forwarded to the archive solution. This is the only way to fulfil the requirements of the law. https://en.wikipedia.org/wiki/Email_archiving
The mailpiler docs tells us:
Postfix
Add the following to main.cf then issue the postfix reload command:

always_bcc = uuid@smtp.example.com

For historical mails there is the import option
https://docs.google.com/document/d/15F0fyb07etMqEXRJnMZcYc016UN-WDds-6zrZNjF6aU/edit#heading=h.58uixy8guf9y

https://docs.google.com/document/d/1YK7zVbcohFWf2w8BpYhCcsAY6Q61JKXa8pqUDrU7nn4/edit#

Both docs are for the enterprise edition. But the oss edition has similar docs.
https://www.mailpiler.org/wiki/current:installation

[timconsidine]

@luckow this makes sense ... except I don't understand why an archive solution would need SMTP for sending. It just needs its own incoming email capability, and the always_bcc means it will get a copy of all inbound and outbound emails from the primary 'working' email server.
But maybe (as is often the case) I am not understanding the full picture.

[luckow]

@timconsidine as I understand it, the smtp port of the archive solution is only relevant for receiving the bcc mails. The user uses the "normal" (in this case the Cloudron) smtp server for sending emails via webmail or the local mail client. And Cloudron smtp forwards/sends any incoming or outgoing mail via smtp to the archive solution. That is the magic in my understanding

[timconsidine]

@luckow thank you
I just didn't know inbound can even be done via SMTP. Either my bad knowledge, or non-standard config.

[luckow]

@timconsidine worth reading https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol

[timconsidine]

@vladimir-d whoop whoop !
Just built, pushed and installed your updated package of Mailpiler.
All went smoothly.
Now I have to set it up and learn it.
Thank you so much !

[marcusquinn]

@timconsidine Nice, more the merrier!

It's definitely another app that can help win mindshare for Cloudron as an enterprise considerate solution with this sort of auditing. Especially when it can be setup as a separate Cloudron instance too for further risk spreading.

[timconsidine]

@marcusquinn agreed.
I totally agree. Have tried some archiving solutions previously and mailpiler looks nice.
But I having a "thick" morning :

• looking through app and their online docs, I can't what address emails should be BCC'ed to. Is it staring me in the face and I can't see it? Do I have to set it up ?

• importing from a server doesn't specify a port number to access IMAP on, so test connection always fails, and it doesn't specify SSL/STARTTLS

I'm embarrassed by these basic questions, but if someone can nudge me in the right direction, I'd appreciate it.

[marcusquinn]

@timconsidine We haven't got to out setup and migration yet, working on 6 things at once, as always! Will bear in mind and answer when we do get to it.

[timconsidine]

@marcusquinn no problem, stumbling through it
Testing some configs
Another Q for later : what is a "mapped domain" when setting up a domain? Can't find any references to it.
No rush.

[marcusquinn]

@jsuto You still with us here? Maybe someone can email him if the forum doesn't?

[necrevistonnezr]

@marcusquinn
BTW does Mailpiler have any sort of compression? I recently tried to switch from the gold standard MailStore Home to another solution (in an attempt to ditch my Windows VM for good) - and my 30 GB MailStore Archive went up to 110 GB(!) in mbox files!

[marcusquinn]

@necrevistonnezr Don't know yet but will post some stats when I get to it. The project I have with it is for years of company email archives, so will have some good stats to compare from source to destination.

I might research file-system level deduplication, as I imagine there's a lot of duplicate files from email threads.

[mazarian]

Just thought I'd chime in and say I'm very interested in a mail archive solution.

[necrevistonnezr]

@marcusquinn said in Mailpiler - self hosted email archive:

@necrevistonnezr Don't know yet but will post some stats when I get to it. The project I have with it is for years of company email archives, so will have some good stats to compare from source to destination.

I might research file-system level deduplication, as I imagine there's a lot of duplicate files from email threads.

Is this still in the works or in a beta stage to be installed?

[marcusquinn]

@necrevistonnezr Hey, we have it working, need to ask @vladimir-d to commit to Cloudron's GitLab and let the team verify it's good to go for App Store inclusion.

[Neiluj]

Hi all,
I am wondering if we are far away from an addition of mailpiler to the Cloudron app repository. It would be of great help in a couple of the situation I am facing atm.
@vladimir-d Maybe you have had the chance to commit the working app version to Clourdon's GitLab already

If I can help (within the remit of my abilities) or contribute in making this happen, I would be happy to do so.

[girish]

I am happy to include this if we have something basic going . @marcusquinn @vladimir-d do you have any initial work here ?

[vladimir.d]

@girish here is some initial work.
https://git.cloudron.io/vladimir.d/mailpiler

TBH I didn't have a chance to complete it yet, I think the issue described here still needs to be addressed.

[savity]

@vladimir-d Would be awesome to get this working... Since the LAW in austria and germany for this hase been activate now for i think 2 Years. That you need a Mail Archive where you can proof the Mail is not changed....

https://www.bfdi.bund.de/DE/Buerger/Inhalte/Finanzen-Steuern/ABC_GoBD.html

[marcusquinn]

@savity As an alternative, you can achieve pretty-much the same thing with EspoCRM, as that will import emails to give you the same duplicate data in an alternative place.

[necrevistonnezr]

@marcusquinn said in Mailpiler - self hosted email archive:

@savity As an alternative, you can achieve pretty-much the same thing with EspoCRM, as that will import emails to give you the same duplicate data in an alternative place.

With full text search of mails and attachments? Can this be used for continuous backup?

[marcusquinn]

@necrevistonnezr full text search of emails, yes. Attachment searching I don't think so but it could be possible as we're using this full-time for 100+ users, so there's some value in us developing anything that isn't there. Sure, just add connection details and it will keep importing emails regardless of if you login or not.

[savity]

@marcusquinn does it import automaticly ? have to check it

[marcusquinn]

@savity Yes, and it's already an Cloudron packaged and maintained app. I recommend it in many ways.

We also have Mailpiler packaged but not yet adopted by the Cloudron team, but have not pursued it further, since Espo is covering our needs.

[jsuto]

Hello, I'm glad to see that there was an initial work adding piler to the app store.
I've got a request recently from a Cloudron user that he wanted to use piler archiving. Can you update me about the current status about this initiative? Whether it's still relevant, or you need some help from my side, etc.

[marcusquinn]

• https://git.cloudron.io/vladimir.d/mailpiler

It works for us, just need adopting by Cloudron with tests and app store inclusion.

[LoudLemur]

@jsuto Thanks for mailpiler. What is the password for the demo? This ought to be written somewhere... more obvious...

[jsuto]

@LoudLemur When you visit https://demo.mailpiler.com/ you may find several account in the username:password format, eg.

admin@local:pilerrocks
auditor@local:auditor
fedora@redhat.com:redhat
external@local:external

Let me know if you have any further questions.

[jsuto]

@marcusquinn It's great news! Can you tell me whom should I contact to progress further?

[marcusquinn]

@jsuto I think that'll be the @appdev Group for community assistance finishing this, or @girish if he's interested in adding it to the App Store.

[jdaviescoates]

@jsuto said in Mailpiler - self hosted email archive:

@marcusquinn It's great news! Can you tell me whom should I contact to progress further?

I guess @staff would just need to give you access to https://git.cloudron.io/vladimir.d/mailpiler

[girish]

@marcusquinn @vladimir-d I have missed this one. Shall I move this to cloudron namespace and publish as unstable?

[marcusquinn]

@girish Sure, it's relatively simple and appears to work as intended. Testing from others would help.

[girish]

@marcusquinn thanks, I can get this published. @vladimir-d can you please put a license?

[timconsidine]

@girish yay ! Thank you.

[vladimir.d]

@girish the License has been added to the repository.
Please feel free to change anything if needed.