-
Hi!
I am reading a lot about how to implement the mail queue at Mautic to manage large amounts of mail sent from the application.Really Mautic's work is sending automated mass mail. By default it is configured to immediately send the emails, but if you send a lot, you can collapse the application.
I found that Rabbitmq is needed on the server, to take care of the mail queue: https://www.rabbitmq.com/download.html
Does anyone expert in this? At the moment I'm going to put this application on the App Wishlist. I need it.
-
-
@girish The queue is useful when you are going to manage many ... many emails that are sent at the same time (it would be our case). If you don't send many, you don't need to use queue management.
Sending a lot of emails causes many emails to be opened at the same time or mautic pages open at the same time, and mautic keeps track of absolutely everything.
If this is not well controlled, Mautic can lose information, including emails.
As you say, it is true that a cron is the way to execute the order to send emails, but if you send many, the best thing is that they are sent through a queue server that acts as a broker, which manages the sending in an orderly way and without losing emails, openings, etc ...
Here is Mautic's documentation about RabbitMQ: https://docs.mautic.org/en/queue
In the Mautic documentation, you will also see that there is another tool called beanstalkd that is also supported by Mautic.
From what I've read, RabbitMQ seems better maintained and ready for docker.
-
I see. That seems fairly heavy duty stuff. I guess a workaround is to use an external RabbitMQ service and configure mautic to use that. i do wonder if all this is required to send lets say 10k mails (then again, I don't know what send many mails means). If it's like 100k or 1 million, I think using more specialized solutions is the way to go.
-
RabbitMQ is a very lightweight queue server and if Mautic has support for it, I think it is because it is necessary if you want to use Mautic on a large scale.
What I'm talking about is that each Mautic application can automate the sending of at least 250,000 emails per month and all the tracking that this generates.
Then multiply this by at least 50 Mautic applications that may be working at the same time (on one or more servers). To do this you need a little queue server that orders the delivery of all emails without missing a single one.
I don't quite understand what you mean by "more specialized solutions." There is no application more specialized in Cloudron than Mautic (well, surely there are some more). It is an application for marketers and businesses with specific needs, and they need sending volume.
However there are other applications in Cloudron that I would never use on a server, such as Minio. I would never use a server as a hard drive in the cloud, it is more expensive than hiring wasabi or google cloud storage.
Just what I like about Cloudron is that I can easily manage specialized applications on my server (or several servers with Cloudron 7) like Mautic, without having to hire external services or external maintenance. I have a @girish man... ;-D
Cloudron makes difficult things easy, and I like that. I would not like this philosophy to change.
We are a Marketing agency and we want to make Mautic reach all possible businesses thanks to Cloudron.
-
@girish said in RabbitMQ for Mautic queue:
If it's like 100k or 1 million, I think using more specialized solutions is the way to go.
@mdreira said in RabbitMQ for Mautic queue:
I don't quite understand what you mean by "more specialized solutions."
I'm assuming @girish just means using specialised email delivery solution like Mailgun, SendGrid, Mailchimp Transactional email, etc. etc.
-
@jdaviescoates We know those and many more... precisely the type of platforms we want to avoid.
-
In my understanding that is a good point to discuss different views on running an app on Cloudron or running it on a dedicated infrastructure. We already had a kind of similar discussion in the field of WebRTC/videoconferencing solutions.
Having Mautic on Cloudron is awesome. You don't need to dig deep into the installation hazzle. You can click on the "install now button" and a few seconds later the Cloudron magic installs everything on your instance. Ideally, the app is perfectly tested and packed, and you will not find any limitations due to a wrong understanding of the app's operation. In this case, my understanding from the Cloudron side is that everything is fine and we (as users) are happy for any update to the app to magically appear on our instances.
If the app works for 10, 100, 1000 or 10.000 users depends on the focus of the app. Having a complete university on a Nextcloud on Cloudron? IMHO not the best idea.
Running Greenlight (as a Frontend for BigBlueButton) with 2.000 users? Maybe it's not the best idea. Running BigBlueButton as an app on Cloudron? Oh no
Running Mautic for small and medium websites with let's say 10.000 visitors a month and a few hundred transactional mails and one or two newsletter to 1000 subscribers? IMHO a good fit. Running the same for a high traffic website with hundreds of time critical transactional mails and thousands of marketing mails? Nope.
This is the moment, as I understand it, when you have to leave the comfort of Cloudron and run a specialised setup for your use case. If you are not able to close the DevOp gap yourself, you need to work with specialised service providers (ping me via DM if you need a partner in that field of experience).
But the good news is, since you have already been able to offer your marketing services to your clients with Mautic on Cloudron, you have been able to gain a lot of experience with the limitations in different projects. And that is important to have good costing bases for your offers.
-
@luckow I do not understand very well why you are limiting the use of cloudron in that way and mautic even more.
As I see it, it depends on how powerful your server or servers you use (the hardware). If Cloudron has enough RAM and CPU, it can manage what you want.
Why do we complicate things? What is the problem?
Rabbitmq is a very simple application and spend very few resources, it is also prepared for Docker. Thanks to this, I do not think this is something as complicated because the Cloudron team can not put it on the application base, such as support for Mautic or other email sending applications like MailTrain.
Mautic is an application made for Marketers and Business, with much more than 1000 subscribers, I assure you.
Do you think you will never use Rabbitmq?. Well, don´t use it. The same thing happens to us with many cloudron applications, and there is no problem for us.
90% of the applications that we will install with Cloudron will be Mautic and WP. I assure you that we will never use the Minecraft server application, File Pizza or Monicahq, but it does not bother me that they are inside Cloudron.
-
Given that the package already uses supervisor, maybe it is worth trying to add a local rabbitmq queue to the app instead of implementing one platform wide. Currently I am not aware of other apps, which may benefit from it, so I guess it makes sense to keep it local to mautic.
I've quickly installed one locally on my system and it indeed appears to be quite lightweight. Maybe the biggest challenge is the backup/restore if it requires special hooks or so. -
@nebulon I think the idea is that there is only one queue server that manages all the emails from all the Mautic applications, that is, something like "an operations center".
If we put a queue server in each Mautic app, I think it would not do its job, since the queue server of a Mautic application would not know what the server of another Mautic application sends ... therefore, we could not control the volume and speed output of all emails that go out together in an orderly way.
-
@mdreira oh I see, wouldn't it then also just work with throttling the mails sent from a mautic instance instead of making the mautic apps all just hand over thousands of mails to some queue? I may not fully get the use-case of the queue then, is the rabbitmq talking to the mail server then to act as some rate-limiting entity? I would expect something to be a consumer of that queue if various mautic instances throw mails into it. Maybe you can shed some more light how it is design to work, I am no marketing mail expert
-
@nebulon Before I comment further on this topic, I want to say something that will surely help you understand why we are actually asking this application in Cloudron.
Right now we are using Amazon SES to send the emails from each Mautic application. Amazon SES has its own queue server (SQS), so we don't urgently need Rabbitmq right now.
As we see a lot of potential in the development of the Cloudron mail server and all the improvements that are going to be made, we think it will be a perfect opportunity to test the Cloudron mail server with the Mautic applications and if it goes well ... to go off using Amazon SES with certain smaller clients.
If Cloudron Mail, together with the rest of the improvements, has a queue server like RabbitMQ that can be configured for all applications that send emails (Mautic, Mailtrain, Freescout, WP ...), the Cloudron mail server would be much more effective and would improve its performance. It could accept much larger volumes of mail.
Just like Amazon SES has a queue server, Cloudron Mail should have one.
I am seeing now that it has been a mistake of mine to request this application only for Mautic, when in fact it is an important improvement for Cloudron Mail. It should be in that section and the title of this post could be changed.
-
@mdreira Thanks for the explanation, I think I understand this feature request a bit better now. What you are desiring is two things - a) making the mail server accept more volumes of mail (if it doesn't already) and b) have a monitorable mail delivery queue. The issue is actually not RabbitMQ as such.
-
So actually from the Mautic documentation: "Processing. Once the queuing mechanism is activated, any page hits and email opens will be queued up to be processed later. To process them, you will need to run some console commands on a regular basis."
It seems to me the queue system for Mautic is not actually related to to email sending, but rather processing email opening or other analytics tasks, which are only stashed away in the queue until the server/mautic has time to process them further. In such a case a per-mautic rabbitmq would solve that just fine.
I feel like we need a better understanding of this first, before throwing in further components. This may or may not be beneficial if running the queue on the same machine.
-
Good catch. It indeed says "Use this if you are getting too much traffic at once from people opening pages or opening emails.". So , the queue is not for email processing, it's really for the callbacks (like 1 pixel images) in the email or if someone clicked on a link in the email.
-
@girish @nebulon Yes! As I wrote above in another comment, Mautic has integration with RabbitmQ to manage tracking hits queue (email openings or page openings), since Rabbitmq supports multiple messaging protocols.
This does not limit Rabbitmq to be used as an email queue server integrated with Cloudron Mail.
If RabbitMQ is integrated into Cloudron Mail, it could manage an orderly queue of all applications that send emails using Cloudron Mail (improving its efficiency and performance) with other benefits for Mautic.
Therefore, I would not install RabbitMQ with Mautic app, but as one more application in Cloudron, which automatically connects with Cloudron Mail, managing the Cloudron Mail queue.
This would make it possible for even another cloudron server that is only used as email server of all applications from other cloudron servers that have a lot of mail sending work (as Mautic). Rabbitmq could connect to different hosts.
-
@mdreira Just reading the docs a bit more, I am not unable to understand how it works.
It seems Mautic uses RabbitMQ to queue the mail instead of sending out the mail instantly. What's not clear is what happens next.
a) Does the mail server directly read from RabbitMQ?
b) Or does mautic read them from RabbitMQ and send to mail sever via SMTP? It seems it is the latter and one has to run
mautic:messages:sendjob to send emails.(If it's b, as a further question, I don't quite get what value the queue adds since all mail servers have a queue of their own anyway. What is the issue with mautic sending mails directly? Generally, a mail queue is useful when the email sending is to be made asynchronous. But isn't mail sending already async in mautic with cron jobs? AFAIK, mails are never sent immediately.)
-
@girish I'm going to try to explain how this would work according to our understanding, because we have never implemented Rabbitmq.
Mautic has a very simple mail queue system, whose sending works with a cronjob. More specifically with the cronjob: mautic:emails:send
Mautic is placing the messages to send within a directory that we set up, and when the cronjob starts, the messages are sent with the speed that we set up.
These settings are in the general Mautic configuration, Email settings section, queue settings: https://prnt.sc/12IL9N1
As you see, you can setup the directory path where you would leave the messages by sending (1), number of messages to be sent (2), time to send them (3) ...
With this, Mautic's workload is reduced a bit if you have thousands of emails to send (since they are being sent bit by bit), but still using Mautic resources with the cron, it does not keep in mind queues of others Mautic apps and delay emails sending.
What we think, is that in the part where the directory route is configured, we could put a directory that is controlled by Rabbitmq and at the same time Rabbitmq is integrated with Cloudron Mail.
We would leave empty the rest of the fields of the Mautic queue configuration that I showed you.
In this way, once Mautic puts the emails in queue, forget them. The Mautic cron "mautic:emails:send" is no longer needed and do not spend more resources, but it is Rabbitmq who manages the sending with Cloudron Mail.
And if we connect all Mautic apps to RabbitmQ, we can asynchronously manage the sending of all emails that are coming from each mautic app by sort and with the configuration we have set. Making the email sending faster and without using Mautic resources. We can also configure a queue of tracking hits, making everything even easier for Mautic.
Therefore, the first test that would have to be done is the integration of Rabbitmq with Cloudron Mail, so that the emails go through the queue before being sent by the mail server.
Then a directory would be configured in Mautic, and that directory in Rabbitmq.
If Cloudron Mail already has a queue system that we can configure by domain (sometimes we can want to prioritize queues of clients more than others) and sends the emails asynchronously, I think it is not necessary Rabbitmq.
The cronjobs do not make email sendings asynchronously at all, because all Mautic's applications have the cron configured in the same minutes of the day.
-
@mdreira Thanks for the detailed response!
I guess step 1 is to make the mail queue visible from the Cloudron Mail side and also have throttle/rate limits and then we have to take it from there. We use Haraka as our mail server and the closest it has for rate limits is http://haraka.github.io/plugins/rate_limit/ . There is no real queue management but there is a status plugin to somewhat monitor it and send commands - http://haraka.github.io/plugins/status/ . Have to look into these further.
