Posts made by d19dotca

The last week or so I've noticed the backups are crashing about 25% of the time, and then most times when invoking the backup manually it works fine again. I've bumped up the backup memory from 4 GB to 6 GB and still see the issues, doesn't seem like the frequency of the issue has gone down at all. The memory (heap) used in the logs seems like there's plenty left, unless I'm reading it incorrectly. If it's memory though, why is it needing more than 2 GB memory increase when it's been very reliable up until the past week or so, without any real changes made in the last week (no new apps or anything like that)?

Any ideas why this may be happening? Logs are below:

Jun 12 08:17:18 box:shell backup-snapshot/box (stdout): 2021-06-12T15:17:18.570Z box:backupupload v8 heap : used 72767888 total: 114642944 max: 5704253440
Jun 12 08:17:19 box:tasks 10692: {"percent":98.36842105263167,"message":"Uploading backup 9301M@14MBps (box)"}
Jun 12 08:17:29 box:tasks 10692: {"percent":98.36842105263167,"message":"Uploading backup 9457M@14MBps (box)"}
Jun 12 08:17:39 box:tasks 10692: {"percent":98.36842105263167,"message":"Uploading backup 9619M@14MBps (box)"}
Jun 12 08:17:48 box:shell backup-snapshot/box (stdout): 2021-06-12T15:17:48.569Z box:backupupload process: rss: 958672896 heapTotal: 110710784 heapUsed: 71391640 external: 298550073
Jun 12 08:17:48 box:shell backup-snapshot/box (stdout): 2021-06-12T15:17:48.569Z box:backupupload v8 heap : used 71397072 total: 110710784 max: 5704253440
Jun 12 08:17:49 box:tasks 10692: {"percent":98.36842105263167,"message":"Uploading backup 9762M@14MBps (box)"}
Jun 12 08:17:59 box:tasks 10692: {"percent":98.36842105263167,"message":"Uploading backup 9915M@14MBps (box)"}
Jun 12 08:18:07 box:shell backup-snapshot/box code: null, signal: SIGKILL
Jun 12 08:18:07 box:taskworker Task took 1086.741 seconds
Jun 12 08:18:07 box:tasks setCompleted - 10692: {"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}
Jun 12 08:18:07 box:tasks 10692: {"percent":100,"result":null,"error":{"stack":"BoxError: Backuptask crashed\n at /home/yellowtent/box/src/backups.js:895:29\n at f (/home/yellowtent/box/node_modules/once/once.js:25:25)\n at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)\n at ChildProcess.emit (events.js:315:20)\n at ChildProcess.EventEmitter.emit (domain.js:467:12)\n at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12)","name":"BoxError","reason":"Internal Error","details":{},"message":"Backuptask crashed"}}

Any help would be appreciated. I'm going to be away tomorrow until Tuesday afternoon so hopefully the backups will be okay in my absence as I won't have internet access.

@robi oh okay, so you’re saying to make a rule that tells all incoming mail to go to a particular new folder then run the other sieve filters on that new folder? I can try later I guess, I’m doubtful that’ll work because that’s kind of the issue… whatever exists that makes spam go to the Spam folder is overwriting everything else, basically completely bypassing Sieve filters. I’ll try when I can though (likely next week, I’m going to be away for the next few days after tomorrow).

@robi hmm, the filter is already “delete all incoming messages” though

@atrilahiji First - congrats! Second - I am unaware of any way to do this. In fact I know we can't change mailbox names, so I guess we can't change our usernames yet either. haha. Hopefully one day both will be possible.

FWIW, I've tried with a general "discard all" after the redirect, but still no-dice which is why I'm certain this is to do with some rule taking a higher priority when it comes to messages marked as spam.

# rule:[Discard messages after redirect complete]
if true
{
	discard;
}

So in my case, the full Sieve file looks like this right now, but still not working to remove any spam messages:

# rule:[Redirect non-spam messages]
if allof (not header :contains "x-spam-status" "Yes")
{
	redirect "<clientExternalEmailAddress>";
}
# rule:[Discard messages after redirect complete]
if true
{
	discard;
}

@robi said in Can't discard spam messages on arrival using Sieve filters:

@d19dotca is it something about the spam folder?

That's for Cloudron team to confirm. I can't say for sure, but yes it certainly seems related to the spam folder itself. If I had to guess, I think the Spam sieve or rules are overriding any of the custom user sieve filters.

can you have spam messages drop in a different folder?

Not that I've been able to do so far but like I said, I haven't tried every possibility. Doesn't seem worthwhile to test every possibility when none of the other ones get to me what I want in my particular use-case anyways. I've done a ton of troubleshooting for various issues, this week I'm a bit tired out. Maybe I'll do more testing next week sometime to get a better picture, but regardless of that I can say for certain that my use-case is not working correctly and I'm pretty certain it's because any user Sieve filters are overwritten by a particular Spam sieve or MTA rule with a higher priority.

can you match on something else in the header and get a log message or indication it did?

As I noted earlier, I can make a rule that works correctly to redirect only non-spam using the sieve filter mentioned in the first comment, so sieve filters definitely work on any message headers including the Spam header when it comes to rules other than discard. Redirect works, for example. But I cannot use the Discard rule apparently on any spam messages.

can you leave the spam in the inbox and discard from there?

It doesn't appear so, no. That's why I suspect this whole thing is being caused by a higher priority Sieve filter or MTA rule that is forcing the message into the Spam folder. There isn't a way that I'm aware of to prevent spam from entering the designated Spam folder and stay in another folder like the inbox for example. This is another reason I'm pretty sure this is all because of a higher priority rule that's preventing any Sieve filters from changing the placement (including discarding completely) any identified spam messages.

can MTA be configured to not deliver spam at all?

That's ultimately what I wanted to some extent. Unfortunately there's no way to prevent spam from being delivered to a mailing list, so that's what led me to consider using mailboxes instead with the sole use-case of redirecting only non-spam and deleting all mail (regardless of spam or not) after it was processed, but no-dice. Wasted a lot of time unfortunately making all these changes and testing. I am one step ahead in the sense that spam isn't forwarded anymore and only non-spam is, but one step back in the sense that now I'm using disk space to store a lot of spam from certain users who receive a ton of it (like talking most days 20-40+ messages a day of just spam because they have a publicly listed email address which has been used for nearly a decade or longer).

does Sieve only work on inbox?

This is a possibility but again, the Cloudron team really needs to confirm this as I am not a programmer so sifting through the code in their Git repo only gets me so far unfortunately. It's their design (whether intentional or not).

@robi That’s the thing, it doesn’t seem like any sieve filters work on messages marked as spam by SpamAssassin when it comes to discarding messages. I had tried a while back the Discard with Reply option and still no dice. I haven’t tested all possibilities, but ultimately I’m needing to discard all messages and it’s not happening if it’s been marked as spam by SpamAssassin before it arrives in the mailbox.

@robi said in Can't discard spam messages on arrival using Sieve filters:

It sounds like you're trying to catch spam before it's marked as spam, then when it is, it gets put into the spam folder correctly, but you don't process it further.

So why not add a sieve rule after it hits the spam folder to delete or only keep X days worth?

No, I don't actually think that’s the issue in this case. If that was the issue then forwarding only non-spam would not work correctly and end up forwarding everything, right? So it’s definitely been processed by SpamAssassin and marked as spam before it hits any of the sieve rules, since the filters I use for sending non-spam only rely on the SpamAssassin headers. By the time any messages hit the sieve filters they’ve already got all of their headers which means they were processed by the different layers already.

Additionally, I’m trying to discard all messages (not just spam messages), I don’t want any messages stored in these mailboxes when they’re dedicated to forwarding only. However only spam messages remain since the redirect to external address works successfully for non-spam - the redirect means it doesn’t save the message locally after forwarding.

@ccfu said in Can't discard spam messages on arrival using Sieve filters:

@d19dotca I tried to set up something similar with filters on Rainloop and can confirm that it does not work as intended - the mails that should be discarded are still delievered to the SPAM folder.

Correct, yeah it’s definitely a sieve thing (not specific to webmail app). I’m disappointed this is the current case because it basically undoes half the reason I had tried migrating from mailing lists to a mailbox in the first place. I always used mailing lists when forwarding all mail to an external address originally sent to a domain hosted on Cloudron, and in order to stop spam from being forwarded on I converted them to mailboxes with sieve rules. Now the spam doesn’t forward on to the target address (this is one step forward), but now it’s storing all the spam mail on disk (this is one step back).

@ccfu said in Is it possible to migrate just emails?:

@d19dotca said in Is it possible to migrate just emails?:

Reason I'm asking is that my understanding is you can't really split roles up between servers for the same domain. In other words, it's not really feasible to have one Cloudron server hosting the website for example.com and another Cloudron server hosting the mail for example.com. My understanding is that everything related to the domain needs to be handled on each server.

Out of interest, why would it not be feasible to split domains between hosting and email? We do this without any problems. On the servers hosting the apps either outbound mail is set to relay via the server handling email or, if the app manages its own email settings, these are set to use the SMTP on the server handling email for the domain.

I may have misunderstood but I think this post is why I was under that impression that the Cloudron domains emails should be on the same server as the apps for that same domain.

https://forum.cloudron.io/topic/5127/problems-with-email-access-on-multi-domain-cloudron

Any help / update on this? I'm wondering @girish - is this perhaps a limitation inside of Cloudron's current configs for Sieve and the webmail apps?

@neurokrish Ports are assigned differently in Docker installs that Cloudron uses. It might help to know what you're trying to access and where you're trying to access it from (i.e. outside of the home network?).

If you're trying to setup access to a home server for access outside of the network, you usually would simply have your router configured with a VPN and DDNS in most cases. You may find something like this useful as well: https://lifehacker.com/geek-to-live-how-to-access-a-home-server-behind-a-rout-127276

You can also use port-forwarding but it's generally just done to the server itself (so the outside ports) rather than individual ports at the app level because those can change and I think are a bit on the dynamic side in Docker depending on the YAML file and other configs.

For incoming ports to Cloudron, you can check out this section in the documentation: https://docs.cloudron.io/security/#inbound-ports

To clarify... you would normally only have your router forward to the server itself on the exposed public ports, since the ports on the actual apps running inside Docker (which is what Cloudron uses for app deployments) are internal ports and not exposed externally.

@dylightful Just looking for some clarification... are you looking to setup a second Cloudron instance on a new server, and have that Cloudron dedicated to mail-only for example? Does the other Cloudron server intend to run apps too?

Reason I'm asking is that my understanding is you can't really split roles up between servers for the same domain. In other words, it's not really feasible to have one Cloudron server hosting the website for example.com and another Cloudron server hosting the mail for example.com. My understanding is that everything related to the domain needs to be handled on each server.

I guess what I'm really saying here is that if your intent is to separate roles/duties between servers, it shouldn't really be done that way and instead should be done in a way where you're separating domains per server.

Just food for thought. But if I've totally misunderstood your use-case, please clarify it and I can try again.

Okay weird update... I changed it to 1G but still received the same error after restarting MySQL service. So then I went back and set it back to 64M default it had (which should have been enough anyways), but then when I tried it again soon after it then suddenly worked importing the JSON file without "Malformed communication packet", it's as if maybe the 1G value worked but wasn't recognized until later? Or it was a total coincidence, I'm not certain which yet. lol.

I'm wondering now if it maybe makes sense to up the max_allowed_packet up a bit though? I mean 64 MB is fine but wondering if it should be closer to 256 MB or something like that instead? I'm not certain if this was even to do with it, but it seemed possible given that it worked soon after tinkering with the values.

Just realized this is probably due to max packet size (max_allowed_packet) in my.cnf for MySQL configuration. I'll test that out later today.

I've been trying to run some tests and was importing my data from Enpass via both json and csv format (Bitwarden supports importing both formats), but I receive the following error no matter which file format I use:

Jun 08 11:45:08 [2021-06-08 18:45:08.405][request][INFO] POST /api/ciphers/import
Jun 08 11:45:08 [2021-06-08 18:45:08.598][error][ERROR] Error saving cipher.
Jun 08 11:45:08 [CAUSE] DatabaseError(
Jun 08 11:45:08 __Unknown,
Jun 08 11:45:08 "Malformed communication packet.",
Jun 08 11:45:08 )
Jun 08 11:45:08 172.18.0.1 - - [08/Jun/2021:18:45:08 +0000] "POST /api/ciphers/import HTTP/1.1" 400 1038 "https://vault.d19.ca/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36"
Jun 08 11:45:08 [2021-06-08 18:45:08.599][response][INFO] POST /api/ciphers/import (post_ciphers_import) => 400 Bad Request

It seems like this is possibly related to MySQL 8, based on some very quick research into Malformed communication packet error text, plus it follows after DatabaseError so it must be MySQL-related somehow.

The CSV is only about 300 KB in size, and the JSON is about 2.7 MB in size (JSON contains the attachments on records too so that's why it's much larger). I don't think it's a size issue though since the CSV is so small and still fails.

Any advice on this? Some configuration change I need to make perhaps?

@girish said in What's coming in Cloudron 6.3:

Having to support 3 ubuntu versions takes a lot of testing time

Maybe it's time to kill off support for Ubuntu 16?

@onlybro That seems more like a phone issue to me, as I don't think apps can change people's networks without notifying them / requesting approval first. Let alone the difference between 3G and 4G which is ultimately using the same cellular connection from a phones point of view.

I am writing a dedicated bug report based on a post I made in another thread.

Use case: Discard spam messages on arrival using Sieve filters via Roundcube.

I was able to setup redirects in the mailbox and they are working where they only forward non-spam messages by using the following filter:

# rule:[Forward non-spam messages]
if allof (not header :contains "x-spam-status" "Yes,")
{
	redirect "<externalEmailAddress>";
}

However, all messages marked as spam still save in the mailbox regardless of any filters I set to discard them. In my case, I want the spam messages to be discarded/deleted automatically as there's no sense in them remaining in the "forwarding only" mailbox. It seems maybe there's a priority filter going on elsewhere in Cloudron that's overriding my sieve filter?

Using a filter like this:

# rule:[Discard spam messages on arrival]
if allof (header :contains "x-spam-status" "Yes,")
{
	discard;
	stop;
}

... will still see the email saved in the Spam folder. I've tried variations of it too such as apply to all messages (i.e. no filter) after the redirect rule is applied, but still no-dice. Basically I can't get any filters to work when it's been identified as spam, only filters when it's NOT spam.

Here's a way to reproduce it:

1. Set this filter in Roundcube:

# rule:[Discard spam messages on arrival]
if allof (header :contains "x-spam-status" "Yes,")
{
	discard;
	stop;
}

2. Send a message to the email account using the GTUBE string which SpamAssassin automatically marks with 1000 points, it's basically a test for spam filters: XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

3. See if the message is really discarded per the sieve filter or if it's still arriving in the Spam folder. In my cases, they still arrive in the Spam folder which seems to be incorrect behaviour.

@girish - this is not possible to do unfortunately in Cloudron it seems, but kindly tell me if I'm doing something wrong here.

The redirect works for filtering out spam, in other words only non-spam messages get through to the endpoint email using a filter like this:

# rule:[Forward non-spam messages]
if allof (not header :contains "x-spam-status" "Yes,")
{
	redirect "<externalEmailAddress>";
}

However, all messages marked as spam still save in the mailbox regardless of any filters I set to discard them. In my case, I want the spam messages to be discarded/deleted automatically as there's no sense in them remaining in the "forwarding only" mailbox. It seems maybe there's a priority filter going on elsewhere in Cloudron that's overriding my sieve filter?

Using a filter like this:

# rule:[Discard spam messages on arrival]
if allof (header :contains "x-spam-status" "Yes,")
{
	discard;
	stop;
}

... will still save the email in the Spam folder. I've tried variations of it too but no-dice. Basically I can't get any filters to work when it's an identified spam message, only filters when it's NOT spam.

Here's a way to reproduce it:

1. Set this filter in Roundcube:

# rule:[Discard spam messages on arrival]
if allof (header :contains "x-spam-status" "Yes,")
{
	discard;
	stop;
}

2. Send a message to the email account using the GTUBE string which SpamAssassin automatically marks with 1000 points, it's basically a test for spam filters: XJS*C4JDBQADN1.NSBN3*2IDNEN*GTUBE-STANDARD-ANTI-UBE-TEST-EMAIL*C.34X

3. See if the message is really discarded per the sieve filter or if it's still arriving in the Spam folder. In my cases, they still arrive in the Spam folder which seems to be incorrect behaviour.

Edit: I wrote a dedicated bug for this instead: https://forum.cloudron.io/topic/5189/can-t-discard-spam-messages-on-arrival-using-sieve-filters/1

I manually ran the Cleanup Backups button on the Backups page, and later in Event Viewer I noticed that despite it removing a couple dozen backups, it states in the main info line that it "removed 0 backups" which seems to be incorrect.

Screenshot:

Full event log entry:

{
  "taskId": "10610",
  "errorMessage": null,
  "removedBoxBackupIds": [
    "2021-05-31-040355-996/box_2021-05-31-043231-075_v6.2.8",
    "2021-05-29-053042-171/box_2021-05-29-055739-509_v6.2.8",
    "2021-05-29-030000-845/box_2021-05-29-032601-218_v6.2.8",
    "2021-05-28-030000-861/box_2021-05-28-032629-295_v6.2.8"
  ],
  "removedAppBackupIds": [
    "2021-05-31-040355-996/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-31-040423-610_v2.11.3",
    "2021-05-31-035911-095/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-31-035937-380_v2.11.3",
    "2021-05-31-035608-888/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-31-035638-030_v2.11.3",
    "2021-05-31-035018-942/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-31-035045-435_v2.11.3",
    "2021-05-31-030000-733/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-31-030033-014_v2.11.3",
    "2021-05-30-070032-089/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-30-070101-985_v2.11.3",
    "2021-05-30-053519-633/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-30-053548-362_v2.11.3",
    "2021-05-30-034742-477/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-30-034812-764_v2.11.3",
    "2021-05-29-053042-171/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-29-053114-674_v2.11.3",
    "2021-05-29-030000-845/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-29-030032-536_v2.11.3",
    "2021-05-28-030000-861/app_134ef7f1-46f8-403e-8ac5-a3297b37893f_2021-05-28-030032-178_v2.11.3",
    "2021-05-31-040355-996/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-31-040419-431_v2.11.3",
    "2021-05-31-035911-095/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-31-035933-138_v2.11.3",
    "2021-05-31-035608-888/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-31-035633-846_v2.11.3",
    "2021-05-31-035018-942/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-31-035040-992_v2.11.3",
    "2021-05-31-030000-733/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-31-030028-195_v2.11.3",
    "2021-05-30-070032-089/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-30-070057-122_v2.11.3",
    "2021-05-30-053519-633/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-30-053542-938_v2.11.3",
    "2021-05-30-034742-477/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-30-034807-541_v2.11.3",
    "2021-05-29-053042-171/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-29-053109-621_v2.11.3",
    "2021-05-29-030000-845/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-29-030027-519_v2.11.3",
    "2021-05-28-030000-861/app_1247a348-c8da-4f30-8a4c-3fa014f092a1_2021-05-28-030027-160_v2.11.3",
    "2021-05-31-040355-996/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-31-040413-556_v2.11.3",
    "2021-05-31-035911-095/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-31-035927-389_v2.11.3",
    "2021-05-31-035608-888/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-31-035628-018_v2.11.3",
    "2021-05-31-035018-942/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-31-035035-121_v2.11.3",
    "2021-05-31-030000-733/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-31-030021-225_v2.11.3",
    "2021-05-30-070032-089/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-30-070049-610_v2.11.3",
    "2021-05-30-053519-633/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-30-053536-951_v2.11.3",
    "2021-05-30-052841-692/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-30-052859-991_v2.11.3",
    "2021-05-30-034742-477/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-30-034801-289_v2.11.3",
    "2021-05-29-053042-171/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-29-053102-826_v2.11.3",
    "2021-05-29-030000-845/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-29-030021-017_v2.11.3",
    "2021-05-28-030000-861/app_05db518f-2c45-411b-865f-813dbf85c244_2021-05-28-030020-704_v2.11.3",
    "2021-05-31-040355-996/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-31-040406-519_v5.16.2",
    "2021-05-31-035911-095/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-31-035921-449_v5.16.2",
    "2021-05-31-035608-888/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-31-035621-564_v5.16.2",
    "2021-05-31-035018-942/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-31-035028-917_v5.16.2",
    "2021-05-31-030000-733/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-31-030014-137_v5.16.2",
    "2021-05-30-070032-089/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-30-070042-797_v5.16.2",
    "2021-05-30-053519-633/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-30-053530-391_v5.16.2",
    "2021-05-30-052841-692/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-30-052853-104_v5.16.2",
    "2021-05-30-034742-477/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-30-034753-384_v5.16.2",
    "2021-05-29-053042-171/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-29-053054-567_v5.16.2",
    "2021-05-29-030000-845/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-29-030012-928_v5.16.2",
    "2021-05-28-030000-861/app_02b2f762-afa4-4278-afaa-5c07ad29715e_2021-05-28-030013-784_v5.16.2",
    "2021-05-31-040355-996/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-31-040405-533_v2.11.3",
    "2021-05-31-035911-095/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-31-035920-603_v2.11.3",
    "2021-05-31-035608-888/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-31-035620-681_v2.11.3",
    "2021-05-31-035018-942/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-31-035027-954_v2.11.3",
    "2021-05-31-030000-733/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-31-030013-006_v2.11.3",
    "2021-05-30-070032-089/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-30-070041-829_v2.11.3",
    "2021-05-30-053519-633/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-30-053529-332_v2.11.3",
    "2021-05-30-052841-692/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-30-052852-079_v2.11.3",
    "2021-05-30-034742-477/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-30-034752-318_v2.11.3",
    "2021-05-29-053042-171/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-29-053053-457_v2.11.3",
    "2021-05-29-030000-845/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-29-030011-856_v2.11.3",
    "2021-05-28-030000-861/app_00895422-a1ff-4196-8bb8-cb4ff8d6eeaa_2021-05-28-030012-661_v2.11.3",
    "2021-05-31-040355-996/app_e18fa18b-b176-4b25-bf78-4d82c022b0f5_2021-05-31-040819-955_v1.6.1",
    "2021-05-29-053042-171/app_e18fa18b-b176-4b25-bf78-4d82c022b0f5_2021-05-29-053540-631_v1.6.1",
    "2021-05-29-030000-845/app_e18fa18b-b176-4b25-bf78-4d82c022b0f5_2021-05-29-030501-053_v1.6.1",
    "2021-05-28-030000-861/app_e18fa18b-b176-4b25-bf78-4d82c022b0f5_2021-05-28-030455-236_v1.6.1",
    "2021-05-31-040355-996/app_f6b21e27-6d25-4c69-a547-0466e622d719_2021-05-31-040932-573_v2.11.3",
    "2021-05-29-053042-171/app_f6b21e27-6d25-4c69-a547-0466e622d719_2021-05-29-053700-301_v2.11.3",
    "2021-05-29-030000-845/app_f6b21e27-6d25-4c69-a547-0466e622d719_2021-05-29-030614-200_v2.11.3",
    "2021-05-28-030000-861/app_f6b21e27-6d25-4c69-a547-0466e622d719_2021-05-28-030608-995_v2.11.3",
    "2021-05-31-040355-996/app_ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9_2021-05-31-040923-660_v2.11.3",
    "2021-05-29-053042-171/app_ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9_2021-05-29-053650-204_v2.11.3",
    "2021-05-29-030000-845/app_ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9_2021-05-29-030604-760_v2.11.3",
    "2021-05-28-030000-861/app_ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9_2021-05-28-030559-908_v2.11.3",
    "2021-05-31-040355-996/app_ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7_2021-05-31-040913-423_v1.21.2",
    "2021-05-29-053042-171/app_ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7_2021-05-29-053639-373_v1.21.2",
    "2021-05-29-030000-845/app_ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7_2021-05-29-030554-925_v1.21.2",
    "2021-05-28-030000-861/app_ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7_2021-05-28-030550-404_v1.21.2",
    "2021-05-31-040355-996/app_e50f78c7-da6e-4036-a1a3-0cc934448cf1_2021-05-31-040853-085_v1.26.2",
    "2021-05-29-053042-171/app_e50f78c7-da6e-4036-a1a3-0cc934448cf1_2021-05-29-053617-272_v1.26.2",
    "2021-05-29-030000-845/app_e50f78c7-da6e-4036-a1a3-0cc934448cf1_2021-05-29-030534-686_v1.26.2",
    "2021-05-28-030000-861/app_e50f78c7-da6e-4036-a1a3-0cc934448cf1_2021-05-28-030530-117_v1.26.2",
    "2021-05-31-040355-996/app_d3121e48-4196-48fe-907d-ee831e11ce5c_2021-05-31-040809-366_v2.1.0",
    "2021-05-29-053042-171/app_d3121e48-4196-48fe-907d-ee831e11ce5c_2021-05-29-053530-494_v2.1.0",
    "2021-05-29-030000-845/app_d3121e48-4196-48fe-907d-ee831e11ce5c_2021-05-29-030450-383_v2.1.0",
    "2021-05-28-030000-861/app_d3121e48-4196-48fe-907d-ee831e11ce5c_2021-05-28-030445-392_v2.1.0",
    "2021-05-31-040355-996/app_c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d_2021-05-31-040752-391_v1.10.11",
    "2021-05-29-053042-171/app_c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d_2021-05-29-053509-965_v1.10.11",
    "2021-05-29-030000-845/app_c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d_2021-05-29-030430-541_v1.10.11",
    "2021-05-28-030000-861/app_c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d_2021-05-28-030429-330_v1.10.11",
    "2021-05-31-040355-996/app_c3ddc268-deef-49e0-8e3a-97e174cf0594_2021-05-31-040750-947_v5.16.2",
    "2021-05-29-053042-171/app_c3ddc268-deef-49e0-8e3a-97e174cf0594_2021-05-29-053508-426_v5.16.2",
    "2021-05-29-030000-845/app_c3ddc268-deef-49e0-8e3a-97e174cf0594_2021-05-29-030429-008_v5.16.2",
    "2021-05-28-030000-861/app_c3ddc268-deef-49e0-8e3a-97e174cf0594_2021-05-28-030427-875_v5.16.2",
    "2021-05-31-040355-996/app_9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd_2021-05-31-040750-171_v5.16.2",
    "2021-05-29-053042-171/app_9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd_2021-05-29-053507-581_v5.16.2",
    "2021-05-29-030000-845/app_9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd_2021-05-29-030428-231_v5.16.2",
    "2021-05-28-030000-861/app_9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd_2021-05-28-030427-040_v5.16.2",
    "2021-05-31-040355-996/app_9f9aba4a-f7b1-44d8-859c-c6953027c39e_2021-05-31-040748-527_v2.11.3",
    "2021-05-29-053042-171/app_9f9aba4a-f7b1-44d8-859c-c6953027c39e_2021-05-29-053504-852_v2.11.3",
    "2021-05-29-030000-845/app_9f9aba4a-f7b1-44d8-859c-c6953027c39e_2021-05-29-030426-586_v2.11.3",
    "2021-05-28-030000-861/app_9f9aba4a-f7b1-44d8-859c-c6953027c39e_2021-05-28-030425-360_v2.11.3",
    "2021-05-31-040355-996/app_95c45c33-bd8a-4fae-9718-509c69456abe_2021-05-31-040740-345_v2.11.3",
    "2021-05-29-053042-171/app_95c45c33-bd8a-4fae-9718-509c69456abe_2021-05-29-053455-470_v2.11.3",
    "2021-05-29-030000-845/app_95c45c33-bd8a-4fae-9718-509c69456abe_2021-05-29-030418-243_v2.11.3",
    "2021-05-28-030000-861/app_95c45c33-bd8a-4fae-9718-509c69456abe_2021-05-28-030415-203_v2.11.3",
    "2021-05-31-040355-996/app_8dba8d47-7890-49d3-8dfa-970e1ec52991_2021-05-31-040734-994_v2.11.3",
    "2021-05-29-053042-171/app_8dba8d47-7890-49d3-8dfa-970e1ec52991_2021-05-29-053448-783_v2.11.3",
    "2021-05-29-030000-845/app_8dba8d47-7890-49d3-8dfa-970e1ec52991_2021-05-29-030411-674_v2.11.3",
    "2021-05-28-030000-861/app_8dba8d47-7890-49d3-8dfa-970e1ec52991_2021-05-28-030407-822_v2.11.3",
    "2021-05-31-040355-996/app_8732aed0-273e-49f5-837f-cd9c600bf7b9_2021-05-31-040729-179_v5.16.2",
    "2021-05-29-053042-171/app_8732aed0-273e-49f5-837f-cd9c600bf7b9_2021-05-29-053442-131_v5.16.2",
    "2021-05-29-030000-845/app_8732aed0-273e-49f5-837f-cd9c600bf7b9_2021-05-29-030405-172_v5.16.2",
    "2021-05-28-030000-861/app_8732aed0-273e-49f5-837f-cd9c600bf7b9_2021-05-28-030401-530_v5.16.2",
    "2021-05-31-040355-996/app_8315f008-87e9-4f62-9ae8-1938d7b3f9b2_2021-05-31-040728-372_v1.2.0",
    "2021-05-29-053042-171/app_8315f008-87e9-4f62-9ae8-1938d7b3f9b2_2021-05-29-053441-345_v1.2.0",
    "2021-05-29-030000-845/app_8315f008-87e9-4f62-9ae8-1938d7b3f9b2_2021-05-29-030404-377_v1.2.0",
    "2021-05-28-030000-861/app_8315f008-87e9-4f62-9ae8-1938d7b3f9b2_2021-05-28-030400-734_v1.2.0",
    "2021-05-31-040355-996/app_76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b_2021-05-31-040725-545_v2.11.3",
    "2021-05-29-053042-171/app_76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b_2021-05-29-053435-721_v2.11.3",
    "2021-05-29-030000-845/app_76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b_2021-05-29-030359-475_v2.11.3",
    "2021-05-28-030000-861/app_76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b_2021-05-28-030354-324_v2.11.3",
    "2021-05-31-040355-996/app_6e21d87b-ea85-420b-8eed-8b14d601438d_2021-05-31-040716-512_v2.11.3",
    "2021-05-29-053042-171/app_6e21d87b-ea85-420b-8eed-8b14d601438d_2021-05-29-053426-773_v2.11.3",
    "2021-05-29-030000-845/app_6e21d87b-ea85-420b-8eed-8b14d601438d_2021-05-29-030350-313_v2.11.3",
    "2021-05-28-030000-861/app_6e21d87b-ea85-420b-8eed-8b14d601438d_2021-05-28-030346-102_v2.11.3",
    "2021-05-31-040355-996/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499_2021-05-31-040705-225_v2.11.3",
    "2021-05-29-053042-171/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499_2021-05-29-053416-369_v2.11.3",
    "2021-05-29-030000-845/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499_2021-05-29-030340-201_v2.11.3",
    "2021-05-28-030000-861/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499_2021-05-28-030335-912_v2.11.3",
    "2021-05-31-040355-996/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-31-040524-091_v5.16.2",
    "2021-05-31-035911-095/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-31-040042-684_v5.16.2",
    "2021-05-31-035608-888/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-31-035744-310_v5.16.2",
    "2021-05-31-035018-942/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-31-035147-003_v5.16.2",
    "2021-05-31-030000-733/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-31-030138-618_v5.16.2",
    "2021-05-30-070032-089/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-30-070212-785_v5.16.2",
    "2021-05-30-053519-633/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-30-053702-553_v5.16.2",
    "2021-05-29-053042-171/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-29-053223-313_v5.16.2",
    "2021-05-29-030000-845/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-29-030147-722_v5.16.2",
    "2021-05-28-030000-861/app_64098702-00d1-4d2e-8d27-4597e100d1c9_2021-05-28-030147-309_v5.16.2",
    "2021-05-31-040355-996/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-31-040523-361_v2.11.3",
    "2021-05-31-035911-095/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-31-040041-890_v2.11.3",
    "2021-05-31-035608-888/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-31-035743-582_v2.11.3",
    "2021-05-31-035018-942/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-31-035146-172_v2.11.3",
    "2021-05-31-030000-733/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-31-030137-789_v2.11.3",
    "2021-05-30-070032-089/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-30-070212-027_v2.11.3",
    "2021-05-30-053519-633/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-30-053701-622_v2.11.3",
    "2021-05-29-053042-171/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-29-053222-523_v2.11.3",
    "2021-05-29-030000-845/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-29-030146-884_v2.11.3",
    "2021-05-28-030000-861/app_633f2c77-9b33-457a-bc3c-919c87d25ca2_2021-05-28-030146-475_v2.11.3",
    "2021-05-31-040355-996/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-31-040515-452_v2.11.3",
    "2021-05-31-035911-095/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-31-040033-209_v2.11.3",
    "2021-05-31-035608-888/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-31-035735-611_v2.11.3",
    "2021-05-31-035018-942/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-31-035138-110_v2.11.3",
    "2021-05-31-030000-733/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-31-030130-250_v2.11.3",
    "2021-05-30-070032-089/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-30-070204-087_v2.11.3",
    "2021-05-30-053519-633/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-30-053650-778_v2.11.3",
    "2021-05-29-053042-171/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-29-053213-401_v2.11.3",
    "2021-05-29-030000-845/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-29-030138-134_v2.11.3",
    "2021-05-28-030000-861/app_6120f84a-0953-4982-9167-ef872c0ee49a_2021-05-28-030137-427_v2.11.3",
    "2021-05-31-040355-996/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-31-040508-862_v2.11.3",
    "2021-05-31-035911-095/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-31-040026-365_v2.11.3",
    "2021-05-31-035608-888/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-31-035728-473_v2.11.3",
    "2021-05-31-035018-942/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-31-035132-077_v2.11.3",
    "2021-05-31-030000-733/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-31-030124-073_v2.11.3",
    "2021-05-30-070032-089/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-30-070156-997_v2.11.3",
    "2021-05-30-053519-633/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-30-053643-238_v2.11.3",
    "2021-05-29-053042-171/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-29-053205-906_v2.11.3",
    "2021-05-29-030000-845/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-29-030130-074_v2.11.3",
    "2021-05-28-030000-861/app_4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e_2021-05-28-030130-367_v2.11.3",
    "2021-05-31-040355-996/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-31-040502-477_v2.11.3",
    "2021-05-31-035911-095/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-31-040019-285_v2.11.3",
    "2021-05-31-035608-888/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-31-035722-429_v2.11.3",
    "2021-05-31-035018-942/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-31-035125-687_v2.11.3",
    "2021-05-31-030000-733/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-31-030116-751_v2.11.3",
    "2021-05-30-070032-089/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-30-070148-485_v2.11.3",
    "2021-05-30-053519-633/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-30-053636-164_v2.11.3",
    "2021-05-29-053042-171/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-29-053159-198_v2.11.3",
    "2021-05-29-030000-845/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-29-030123-427_v2.11.3",
    "2021-05-28-030000-861/app_493c4f88-5d4d-40e5-8361-fd7d4541441d_2021-05-28-030123-468_v2.11.3",
    "2021-05-31-040355-996/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-31-040456-693_v1.18.2",
    "2021-05-31-035911-095/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-31-040011-092_v1.18.2",
    "2021-05-31-035608-888/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-31-035715-965_v1.18.2",
    "2021-05-31-035018-942/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-31-035119-599_v1.18.2",
    "2021-05-31-030000-733/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-31-030108-979_v1.18.2",
    "2021-05-30-070032-089/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-30-070138-264_v1.18.2",
    "2021-05-30-053519-633/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-30-053627-995_v1.18.2",
    "2021-05-29-053042-171/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-29-053151-513_v1.18.2",
    "2021-05-29-030000-845/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-29-030115-234_v1.18.1",
    "2021-05-28-030000-861/app_492b4264-fea8-4d52-bbe6-1f15ce72c53a_2021-05-28-030114-989_v1.18.1",
    "2021-05-31-040355-996/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-31-040455-422_v2.11.3",
    "2021-05-31-035911-095/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-31-040009-380_v2.11.3",
    "2021-05-31-035608-888/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-31-035713-030_v2.11.3",
    "2021-05-31-035018-942/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-31-035118-340_v2.11.3",
    "2021-05-31-030000-733/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-31-030107-668_v2.11.3",
    "2021-05-30-070032-089/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-30-070136-969_v2.11.3",
    "2021-05-30-053519-633/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-30-053626-638_v2.11.3",
    "2021-05-30-034742-477/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-30-034847-610_v2.11.3",
    "2021-05-29-053042-171/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-29-053150-231_v2.11.3",
    "2021-05-29-030000-845/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-29-030113-658_v2.11.3",
    "2021-05-28-030000-861/app_4929b73d-cfa4-4246-82a7-4bc8127982f6_2021-05-28-030113-527_v2.11.3",
    "2021-05-31-040355-996/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-31-040447-138_v5.16.2",
    "2021-05-31-035911-095/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-31-040000-702_v5.16.2",
    "2021-05-31-035608-888/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-31-035703-856_v5.16.2",
    "2021-05-31-035018-942/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-31-035110-168_v5.16.2",
    "2021-05-31-030000-733/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-31-030059-528_v5.16.2",
    "2021-05-30-070032-089/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-30-070127-348_v5.16.2",
    "2021-05-30-053519-633/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-30-053617-085_v5.16.2",
    "2021-05-30-034742-477/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-30-034838-175_v5.16.2",
    "2021-05-29-053042-171/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-29-053140-637_v5.16.2",
    "2021-05-29-030000-845/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-29-030104-039_v5.16.2",
    "2021-05-28-030000-861/app_40ae3d28-d67f-453e-895d-0f22b539372c_2021-05-28-030103-689_v5.16.2",
    "2021-05-31-040355-996/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-31-040446-431_v1.7.0",
    "2021-05-31-035911-095/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-31-035959-948_v1.7.0",
    "2021-05-31-035608-888/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-31-035703-084_v1.7.0",
    "2021-05-31-035018-942/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-31-035109-398_v1.7.0",
    "2021-05-31-030000-733/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-31-030058-819_v1.7.0",
    "2021-05-30-070032-089/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-30-070126-318_v1.7.0",
    "2021-05-30-053519-633/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-30-053616-291_v1.7.0",
    "2021-05-30-034742-477/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-30-034837-464_v1.7.0",
    "2021-05-29-053042-171/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-29-053139-756_v1.7.0",
    "2021-05-29-030000-845/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-29-030103-274_v1.7.0",
    "2021-05-28-030000-861/app_38f883dc-4047-4237-a6a1-3ecd77efcde9_2021-05-28-030102-847_v1.7.0",
    "2021-05-31-040355-996/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-31-040445-270_v2.5.1",
    "2021-05-31-035911-095/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-31-035958-738_v2.5.1",
    "2021-05-31-035608-888/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-31-035701-796_v2.5.1",
    "2021-05-31-035018-942/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-31-035107-988_v2.5.1",
    "2021-05-31-030000-733/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-31-030057-475_v2.5.1",
    "2021-05-30-070032-089/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-30-070124-946_v2.5.1",
    "2021-05-30-053519-633/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-30-053614-785_v2.5.1",
    "2021-05-30-034742-477/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-30-034836-125_v2.5.1",
    "2021-05-29-053042-171/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-29-053138-421_v2.5.1",
    "2021-05-29-030000-845/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-29-030101-940_v2.5.1",
    "2021-05-28-030000-861/app_3167ef7d-e4c2-4072-8e11-8de49c5db970_2021-05-28-030101-347_v2.5.1",
    "2021-05-31-040355-996/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-31-040444-448_v2.11.3",
    "2021-05-31-035911-095/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-31-035957-891_v2.11.3",
    "2021-05-31-035608-888/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-31-035700-852_v2.11.3",
    "2021-05-31-035018-942/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-31-035107-059_v2.11.3",
    "2021-05-31-030000-733/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-31-030056-563_v2.11.3",
    "2021-05-30-070032-089/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-30-070124-003_v2.11.3",
    "2021-05-30-053519-633/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-30-053613-745_v2.11.3",
    "2021-05-30-034742-477/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-30-034835-148_v2.11.3",
    "2021-05-29-053042-171/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-29-053137-478_v2.11.3",
    "2021-05-29-030000-845/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-29-030100-923_v2.11.3",
    "2021-05-28-030000-861/app_29354a9c-7e1d-4986-b78e-3510fe73b370_2021-05-28-030100-393_v2.11.3",
    "2021-05-31-040355-996/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-31-040437-448_v2.11.3",
    "2021-05-31-035911-095/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-31-035951-460_v2.11.3",
    "2021-05-31-035608-888/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-31-035653-501_v2.11.3",
    "2021-05-31-035018-942/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-31-035059-912_v2.11.3",
    "2021-05-31-030000-733/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-31-030049-673_v2.11.3",
    "2021-05-30-070032-089/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-30-070117-237_v2.11.3",
    "2021-05-30-053519-633/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-30-053605-316_v2.11.3",
    "2021-05-30-034742-477/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-30-034828-055_v2.11.3",
    "2021-05-29-053042-171/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-29-053130-127_v2.11.3",
    "2021-05-29-030000-845/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-29-030051-640_v2.11.3",
    "2021-05-28-030000-861/app_21aecd6a-398a-433a-a954-5bcd5f861e20_2021-05-28-030051-375_v2.11.3",
    "2021-05-31-040355-996/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-31-040430-000_v2.11.3",
    "2021-05-31-035911-095/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-31-035943-468_v2.11.3",
    "2021-05-31-035608-888/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-31-035644-946_v2.11.3",
    "2021-05-31-035018-942/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-31-035051-986_v2.11.3",
    "2021-05-31-030000-733/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-31-030041-096_v2.11.3",
    "2021-05-30-070032-089/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-30-070107-787_v2.11.3",
    "2021-05-30-053519-633/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-30-053555-881_v2.11.3",
    "2021-05-30-034742-477/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-30-034819-721_v2.11.3",
    "2021-05-29-053042-171/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-29-053121-449_v2.11.3",
    "2021-05-29-030000-845/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-29-030040-821_v2.11.3",
    "2021-05-28-030000-861/app_200d36ca-8109-409a-9b56-07f46c2f09bd_2021-05-28-030040-493_v2.11.3",
    "2021-05-28-030000-861/app_ca7827f8-4686-4818-a821-f92fe4f5726d_2021-05-28-030430-473_v1.0.8"
  ],
  "missingBackupIds": []
}

I can update that the first rule doesn't seem to work in my sieve filter, it's not discarding the spam messages but is still saving them in spam instead. However the main use-case of only forwarding on non-spam messages seems to work well. I'm still trying to figure out the first one, because basically I don't want any email saved in the mailbox when it's being used for forwarding only.

Local / external block storage disks aren't often "cost effective", but definitely offer the most reliability and best performance. If your priority is cost-effective, then yes you'd need to probably stick to S3-compliant services. Personally I've used OVH Object Storage which is s3-compliant and works well, but certainly isn't the best performance-wise.

Currently I'm using Vultr's Object Storage and it's a bit more costly than OVH for example (especially after I convert the currency) but it's also far more performant. For example, I'd get maybe an average transfer rate of about 5 Mbps in OVH (range was from about 1 Mbps to 15 Mbps), but I get pretty consistently about 15 Mbps at Vultr (range I've seen is from about 8 Mbps to 40 Mbps). It's one of those "you get what you pay for" scenarios generally when it comes to object storage like that, at least in my own personal experiences so far.

@vladimir What other configurations are needed by redisearch though? I think it needs to be laid out specifically so that the amazing Cloudron team can figure out what can be safely added if any to the current redis server configuration, or if it's configurations needed within the WordPress server/app instead, etc. When you deploy redissearch, does it error on anything, what errors does it provide in the user interface and the logs for example?

Okay, so I ran another test and I think this makes more sense now and sort of validates what you mentioned above.

I spun up a new VPS on Vultr on Ubuntu 20.04 (not the Cloudron marketplace app version), and before I did anything, I immediately uninstalled landscape. This took away UID 107. Then I installed Cloudron, and of course Cloudron created the mysql user with UID 107 which can be verified in /etc/password file too.

Here's my current server:

tcpdump:x:105:111::/nonexistent:/usr/sbin/nologin
sshd:x:106:65534::/run/sshd:/usr/sbin/nologin
pollinate:x:108:1::/var/cache/pollinate:/bin/false
systemd-network:x:109:114:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:110:115:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
systemd-timesync:x:111:116:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
mysql:x:112:118:MySQL Server,,,:/nonexistent:/bin/false
unbound:x:113:119::/var/lib/unbound:/usr/sbin/nologin
nginx:x:114:120:nginx user,,,:/nonexistent:/bin/false
yellowtent:x:1000:1000::/home/yellowtent:/bin/sh

Notice MySQL took UID 112 since at the time of install, landscape user was already generated with UID 107.

On a new server install where I purged landscape prior to installing Cloudron, the MySQL user then takes UID 107 since it's "available":

tcpdump:x:105:111::/nonexistent:/usr/sbin/nologin
sshd:x:106:65534::/run/sshd:/usr/sbin/nologin
pollinate:x:108:1::/var/cache/pollinate:/bin/false
systemd-network:x:109:114:systemd Network Management,,,:/run/systemd:/usr/sbin/nologin
systemd-resolve:x:110:115:systemd Resolver,,,:/run/systemd:/usr/sbin/nologin
systemd-timesync:x:111:116:systemd Time Synchronization,,,:/run/systemd:/usr/sbin/nologin
systemd-coredump:x:999:999:systemd Core Dumper:/:/usr/sbin/nologin
mysql:x:107:113:MySQL Server,,,:/nonexistent:/bin/false
unbound:x:112:118::/var/lib/unbound:/usr/sbin/nologin
nginx:x:113:119:nginx user,,,:/nonexistent:/bin/false
yellowtent:x:1000:1000::/home/yellowtent:/bin/sh

So when I look at the top output, there's no UID being shown anymore and basically everything appears to run as the mysql user since it's UID 107 which then matches the UID used in the container images:

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                               
    614 mysql     20   0 1069132 141280  35112 S   0.0  14.1   0:01.01 mysqld                                
    643 root      20   0  725468  92732  49580 S   0.0   9.2   0:00.35 dockerd                               
    935 yellowt+  20   0  638720  61260  30140 S   0.0   6.1   0:01.11 node                                  
    521 root      20   0  749848  46900  26172 S   0.0   4.7   0:00.32 containerd                            
    497 yellowt+  20   0  579328  34044  27896 S   0.0   3.4   0:00.16 node                                  
    579 root      20   0  110808  21048  13392 S   0.0   2.1   0:00.07 unattended-upgr                       
    504 root      20   0   31976  18176  10464 S   0.0   1.8   0:00.09 networkd-dispat  
[...]

So I guess if I want to "clean" this up (even though there's really no issue at all I guess outside of there being no associated username with UID 107), I should reimage my server and restore from the backup after installing Cloudron only after I've already purged landscape. Though this I admit is really not necessary and I should just get used to seeing UID 107 in the process list and top output and I should know that's coming from the container UIDs then rather than the actual account in Ubuntu (outside of the fact when it actually is MySQL running, lol).

Hopefully the above makes some sense.

@robi So in other words, you're wanting Cloudron to essentially notify you if it notices that registered domain names used on the Cloudron instance are pointed away from the server? In other words, a period check (like once a day or something) that notifies admins?

@girish Yeah that's in Vultr though and what my latest experience has been too. But I guess what I'm meaning is when I have used CLoudron on other providers like LunaNode, OVH, etc. I've never seen this issue before. I suspect it's because there is no UID 107 in the Ubuntu OS in those provider's implementations of it, but if that were the case then I'd supposedly see "107" in all my top output which I've never ever noticed before. That's what I want to try and figure out next, curiosity is getting the better of me. haha. This has been a very interesting puzzle and learning experience.

@girish Ahhh okay, haha, than I guess that makes sense. What a bizarre thing, lol. So if I understand correctly, the next UID in your service containers will get UID 107 since the Docker images you base from go up to UID 106 already. And in my case, since Vultr has it's Ubuntu image using landscape as UID 107 on the operating system level, that's why it looks all weird for me. Okay, I think that makes more sense to me then. haha.

Though one last question... if there was no UID 107 in the /etc/passwd file for example (which I confirmed last night is the case with OVH's instance of Ubuntu), then when Cloudron sets its containers and its user is the given UID 107 in the container, why does top not show 107 there instead since there's no system UID 107? I should investigate that a bit more, I think that's the last part of my curiosity. haha.

Thanks for bearing with me and explaining everything Girish! I really appreciate it. Always love learning from the experts on these things.

@girish haha, sorry, I was kind of just brain dumping as I learned more about it myself, didn't mean to make it extra confusing.

Okay so I understood how UIDs on the Ubuntu system level work, but I guess where I'm confused is how Cloudron specifies a UID for it's main process in each container, and why are they all the same (in my case they're all UID 107 in each service container, is it always 107 even in new installs?). I think that's where it's throwing me off. Why is Cloudron using UID 107 for example in all of it's service containers running user (like mysql, dovecot, postgresql, etc)?

Additionally, I never have seen a UID used when looking at top before, so is there something perhaps not "registered" properly in my install?

As I understand it, Cloudron's containers - at least in my case - are using it's main user accounts for various services with UID 107. In Ubuntu, UID 107 happened to match the landscape user. So when I uninstalled landscape which removed the user too, now there's no system level UID 107 in Ubuntu, but I guess top sees the UID from the container as UID 107 running the process like mysql, so it outputs 107.

I have just never seen this before in any of my many other servers I've build with Cloudron.... so I'm confused I guess by why this is happening in this server seemingly alone, where UID 107 gets generated from in Cloudron's container services, etc.

Hopefully that helps a little bit clarify where my confusion is and what I'm hoping to understand better.

@nebulon said in Safe to truncate /home/yellowtent/platformdata/logs/* when large disk consumer?:

@d19dotca is it possible that the missing logrotate files would be for already uninstalled apps and those large log files are just not cleaned up after uninstallation?

No, unfortunately not. That has never been the case either, wish it was that easy. haha. I reported earlier in this thread that re-saving an app with from Location tab will regenerate the logrotate configs for the app. They just go missing randomly, no idea why they disappear, but I at least know how to get them back. Unfortunately I am not aware of any of this until I randomly check sizing on my server and find incredibly large log files which then leads me to discover the apps are missing logrotate configs. Saving the app again so it sort of rebuilds itself will regenerate it. I just looked now and there are a bunch more, for example: I didn't do anything beyond restoring the Cloudron backup to a new Cloudron instance in the Vultr Marketplace last night:

ls -alh /home/yellowtent/platformdata/logrotate.d
total 156K
drwxr-xr-x  2 root       root       4.0K May 31 03:54 .
drwxr-xr-x 15 yellowtent yellowtent 4.0K May 30 06:27 ..
-rw-r--r--  1 root       root        946 May 30 06:28 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
-rw-r--r--  1 root       root        946 May 30 06:28 02b2f762-afa4-4278-afaa-5c07ad29715e
-rw-r--r--  1 root       root        946 May 30 06:28 05db518f-2c45-411b-865f-813dbf85c244
-rw-r--r--  1 root       root        946 May 30 06:28 1247a348-c8da-4f30-8a4c-3fa014f092a1
-rw-r--r--  1 root       root        946 May 30 06:28 134ef7f1-46f8-403e-8ac5-a3297b37893f
-rw-r--r--  1 root       root        946 May 30 06:28 200d36ca-8109-409a-9b56-07f46c2f09bd
-rw-r--r--  1 root       root        946 May 30 06:28 21aecd6a-398a-433a-a954-5bcd5f861e20
-rw-r--r--  1 root       root        946 May 30 06:28 29354a9c-7e1d-4986-b78e-3510fe73b370
-rw-r--r--  1 root       root        946 May 30 06:28 3167ef7d-e4c2-4072-8e11-8de49c5db970
-rw-r--r--  1 root       root        946 May 30 06:29 38f883dc-4047-4237-a6a1-3ecd77efcde9
-rw-r--r--  1 root       root        946 May 30 06:28 40ae3d28-d67f-453e-895d-0f22b539372c
-rw-r--r--  1 root       root        946 May 30 06:29 4929b73d-cfa4-4246-82a7-4bc8127982f6
-rw-r--r--  1 root       root        946 May 30 06:29 492b4264-fea8-4d52-bbe6-1f15ce72c53a
-rw-r--r--  1 root       root        946 May 30 06:29 493c4f88-5d4d-40e5-8361-fd7d4541441d
-rw-r--r--  1 root       root        946 May 30 06:29 4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e
-rw-r--r--  1 root       root        946 May 30 06:29 6120f84a-0953-4982-9167-ef872c0ee49a
-rw-r--r--  1 root       root        946 May 30 06:29 633f2c77-9b33-457a-bc3c-919c87d25ca2
-rw-r--r--  1 root       root        946 May 30 06:29 64098702-00d1-4d2e-8d27-4597e100d1c9
-rw-r--r--  1 root       root        946 May 31 03:54 6b3ebc4f-9708-4243-bc21-4c9a01bc3499
-rw-r--r--  1 root       root        946 May 30 06:29 6e21d87b-ea85-420b-8eed-8b14d601438d
-rw-r--r--  1 root       root        946 May 30 06:30 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
-rw-r--r--  1 root       root        946 May 30 06:29 8315f008-87e9-4f62-9ae8-1938d7b3f9b2
-rw-r--r--  1 root       root        946 May 30 06:29 8732aed0-273e-49f5-837f-cd9c600bf7b9
-rw-r--r--  1 root       root        946 May 30 06:30 8dba8d47-7890-49d3-8dfa-970e1ec52991
-rw-r--r--  1 root       root        946 May 30 06:30 95c45c33-bd8a-4fae-9718-509c69456abe
-rw-r--r--  1 root       root        946 May 30 06:30 9f9aba4a-f7b1-44d8-859c-c6953027c39e
-rw-r--r--  1 root       root        946 May 30 06:30 9fdedb7f-1bd4-42f6-a7e8-9831df9da8dd
-rw-r--r--  1 root       root        230 May 28 17:54 box
-rw-r--r--  1 root       root        946 May 30 06:30 c3ddc268-deef-49e0-8e3a-97e174cf0594
-rw-r--r--  1 root       root        946 May 30 06:31 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
-rw-r--r--  1 root       root        946 May 30 06:32 d3121e48-4196-48fe-907d-ee831e11ce5c
-rw-r--r--  1 root       root        946 May 30 06:32 e18fa18b-b176-4b25-bf78-4d82c022b0f5
-rw-r--r--  1 root       root        946 May 30 06:33 e50f78c7-da6e-4036-a1a3-0cc934448cf1
-rw-r--r--  1 root       root        946 May 30 06:31 ea2ee30a-8d3b-4a3b-8817-19b0f1e17bd7
-rw-r--r--  1 root       root        946 May 30 06:31 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
-rw-r--r--  1 root       root        946 May 30 06:31 f6b21e27-6d25-4c69-a547-0466e622d719
-rw-r--r--  1 root       root       1.1K May 28 17:54 platform

Okay so here's my theory of what's happened before, would love it if @girish could sort of sanity check this for me.

In previous installs, there was no UID 107 used either in Ubuntu or Cloudron services. When I tried a default install in OVH where I didn't ever see landscape before, it turns out landscape DID exist but was never seen in top because it had a UID of 113 or something like that, not 107.

Since in my case, the Vultr Ubuntu image has landscape as UID 107 by default, so all the container services running as UID 107 (but mysql, postreqsql, dovecot, etc) would then appear to be run by Lanscape user when it really was just a mix of different users all with the same UID of 107.

Thus when I removed landscape which also removed the user that was mapped to UID 107, then since the Cloudron services are coincidentally using UID 107, top output now shows user 107.

Curious though... where does UID 107 come from? I.e. How did Cloudron pick that UID, and why is that UID shared among all the containers but using different usernames in the containers with that UID?

Is my current setup best practice? Is there any issue with there only being a UID shown and no associated username? Would this be resolved if I reimaged the appliance, and then removed landscape prior to even starting the Cloudron install (last time I did it a few minutes during the install so I wonder if I did it too late), would it then create a user properly for that? I tried briefly to do that and then install Cloudron on a new smaller VPS in Vultr, and it seemed like it created MySQL user at 107 instead or something, so the MySQL user in Ubuntu was running many of the services for Cloudron. But I should probably test again.

I've never seen a UID before at all in my Cloudron servers which aren't associated to any username when looking at top, so I feel like this is just a very unusual circumstance in that Cloudron is running its own services with UID 107 in containers, but Ubuntu image in Vultr had UID 107 associated to Landscape, causing the confusions.

Hopefully the above makes sense. Would love your insight into this.

@robi said in Performance Tuning:

As per: https://linuxhint.com/understanding_vm_swappiness/

Append to bottom of /etc/sysctl.conf

# VM Swappiness
vm.swappiness = 10

Did you find any noticeable performance improvement by setting that property? I'm looking into fine-tuning my Ubuntu image.

Nevermind, lol. I increased the memory further to 7 GB while also lowering the message size from 1 GB to 512 MB, and suddenly it's all working again. What's weird to me is this was never an issue before, and this app that kept triggering the fail was always on my server since basically day one of starting Cloudron years ago. Also my server's backups were successful now for about a month or two since the last (unrelated) incident. No new apps installed, if anything it's one or two less now. So no idea what would trigger that. haha but it seems to be okay now.

By the way, since there's a ton of backup crashing that seem memory related... is it possible to improve the logging so that there's something in there saying "OutOfMemory" or something like that so poeple know why it's crashing?

I've noticed one app is failing to backup and causing the full backup task to crash. I thought it was memory related so I bumped it up from 4 GB to 6 GB max for backup process, same issue though. What's odd is that if I backup the app individually, it succeeds. So I'm not certain what's causing the issue.

ay 30 21:01:13 box:tasks 10577: {"percent":52.35135135135134,"message":"Uploading backup 806M@26MBps (<appSubDomain>)"}
May 30 21:01:24 box:shell backup-snapshot/app_6b3ebc4f-9708-4243-bc21-4c9a01bc3499 code: null, signal: SIGKILL
May 30 21:01:24 box:backups <appSubDomain> Unable to backup BoxError: Backuptask crashed
at /home/yellowtent/box/src/backups.js:895:29
at f (/home/yellowtent/box/node_modules/once/once.js:25:25)
at ChildProcess.<anonymous> (/home/yellowtent/box/src/shell.js:69:9)
at ChildProcess.emit (events.js:315:20)
at ChildProcess.EventEmitter.emit (domain.js:467:12)
at Process.ChildProcess._handle.onexit (internal/child_process.js:277:12) {
reason: 'Internal Error',
details: {}
May 30 21:01:24 box:taskworker Task took 133.496 seconds

Any suggestions?

@robi yea that’d be awesome! Somewhat related to my other issue where the UID used by Cloudron containers was totally confusing me.

Okay digging into it further, I dropped into bash prompts for other containers like mail, postgresql, etc. And the main service ID in all of them is 107. So I guess this UID 107 when viewing ps or too makes sense? I’m just still confused because I’ve never noticed that behaviour before in the last two or so years of using Cloudron. Why is UID 107 chosen for all containers? How does that UID get set? Also why doesn’t the process list see that user? I swear I never had UIDs shown before in previous installs of Cloudron. I can’t wrap my head around this. Lol. But I admit I guess the landscape packages are indeed removed then, and I guess it’s a coincidence that the main user ID for each Cloudron service container is 107?

So I’m definitely confused. When I spin up a new image, while Landscape is installed it isn’t actively used (I can’t find any running processes with that user), but it definitely gets triggered later when Cloudron is running. I reimaged my server last night to the new Cloudron marketplace app in Vultr, removed landscape before restoring, but eventually ended up in the exact same spot of landscape uninstalled but it’s UID 107 still running (as a UID only) it’s processes as it did earlier when it was installed. It’s the weirdest thing.

Interestingly, when I tempoprarily spun up a new Cloudron image using the Vultr Marketplace, and if I removed the landscape-common first thing, then it seemed the MySQL instances and everything were only seen in top with the MySQL user, no landscape or 107 seen. So I'm going to try spinning up a new permanent Cloudron server using the Cloudron app image on Vultr Marketplace and restore using Cloudron backups. I hope this will clean things up better with any luck.

The logs are still being poorly maintained it seems, they account for several GBs of disk space which seems crazy high for just logs:

813M	/home/yellowtent/platformdata/logs/6b3ebc4f-9708-4243-bc21-4c9a01bc3499/app.log
788M	/home/yellowtent/platformdata/logs/4929b73d-cfa4-4246-82a7-4bc8127982f6/app.log
712M	/home/yellowtent/platformdata/logs/6e21d87b-ea85-420b-8eed-8b14d601438d/app.log
656M	/home/yellowtent/platformdata/logs/493c4f88-5d4d-40e5-8361-fd7d4541441d/app.log
631M	/home/yellowtent/platformdata/logs/6120f84a-0953-4982-9167-ef872c0ee49a/app.log
629M	/home/yellowtent/platformdata/logs/f6b21e27-6d25-4c69-a547-0466e622d719/app.log
613M	/home/yellowtent/platformdata/logs/4d2fc2b7-2475-4c6b-97b4-4f896fbe8e9e/app.log
596M	/home/yellowtent/platformdata/logs/21aecd6a-398a-433a-a954-5bcd5f861e20/app.log
578M	/home/yellowtent/platformdata/logs/05db518f-2c45-411b-865f-813dbf85c244/app.log
514M	/home/yellowtent/platformdata/logs/8dba8d47-7890-49d3-8dfa-970e1ec52991/app.log

In my case, it seems the same issue I reported back February remains... there are missing logrotate.conf files for the area where there's far too high of log file sizes.

Example: Note that logs for 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 was the largest... when I run the following command, there's nothing returned, suggesting the logrotate.conf is missing for this one, and likely all the rest too: ls -alh /home/yellowtent/platformdata/logrotate.d/ | grep 6b3ebc4f-9708-4243-bc21-4c9a01bc3499 returns no files.

When I do an ls -alh listing for the directory, here is all that exists which seems far too few:

ls -alh /home/yellowtent/platformdata/logrotate.d
total 80K
drwxr-xr-x  2 root       root       4.0K May 29 05:00 .
drwxr-xr-x 15 yellowtent yellowtent 4.0K May 15 07:27 ..
-rw-r--r--  1 root       root        946 May 15 20:21 00895422-a1ff-4196-8bb8-cb4ff8d6eeaa
-rw-r--r--  1 root       root        946 May 17 22:43 02b2f762-afa4-4278-afaa-5c07ad29715e
-rw-r--r--  1 root       root        946 May 20 16:28 200d36ca-8109-409a-9b56-07f46c2f09bd
-rw-r--r--  1 root       root        946 May 21 17:35 29354a9c-7e1d-4986-b78e-3510fe73b370
-rw-r--r--  1 root       root        946 May 27 05:00 3167ef7d-e4c2-4072-8e11-8de49c5db970
-rw-r--r--  1 root       root        946 May 29 05:00 492b4264-fea8-4d52-bbe6-1f15ce72c53a
-rw-r--r--  1 root       root        946 May 20 16:23 633f2c77-9b33-457a-bc3c-919c87d25ca2
-rw-r--r--  1 root       root        946 May 17 22:41 64098702-00d1-4d2e-8d27-4597e100d1c9
-rw-r--r--  1 root       root        946 May 23 06:28 76aa4cfd-a1c7-451a-a11e-a4fabe5ff95b
-rw-r--r--  1 root       root        946 May 16 06:37 9f9aba4a-f7b1-44d8-859c-c6953027c39e
-rw-r--r--  1 root       root        230 May 15 09:00 box
-rw-r--r--  1 root       root        946 May 17 22:38 c3ddc268-deef-49e0-8e3a-97e174cf0594
-rw-r--r--  1 root       root        946 May 27 20:55 c40f9f5f-037c-4a1d-8538-f1f4d4b82f5d
-rw-r--r--  1 root       root        946 May 27 21:00 ca7827f8-4686-4818-a821-f92fe4f5726d
-rw-r--r--  1 root       root        946 May 18 05:00 e18fa18b-b176-4b25-bf78-4d82c022b0f5
-rw-r--r--  1 root       root        946 May 26 15:44 e50f78c7-da6e-4036-a1a3-0cc934448cf1
-rw-r--r--  1 root       root        946 May 24 00:24 ed41aa60-ff4d-4e8e-8796-8c7de4bdf1a9
-rw-r--r--  1 root       root       1.1K May 15 09:00 platform

@girish said in Ubuntu 20.04 "landscape" user account running mysqld:

@d19dotca atleast you removed unused packages the id inside the container is 107 (mysqld). you can do docker exec -ti mysql /bin/bash and do id mysql that will be 107.

Very interesting, but now I'm a bit confused, lol. I realize though this is probably a bit outside the realm of Cloudron so no worries if you can't really explain it. Here's what I've seen, if you're able to somehow make heads or tails of this, I'd really appreciate your insight:

When I ran the docker exec -ti mysql /bin/bash and then id mysql, you are correct, the UID is 107, as seen below from the container commands output:

root@mysql:/# id mysql
uid=107(mysql) gid=109(mysql) groups=109(mysql)

But what I'm confused on is that the output prior was also UID 107 for the landscape user in the Ubuntu operating system (not in any containers):

id landscape
uid=107(landscape) gid=113(landscape) groups=113(landscape)

And the MySQL user in the Ubuntu operating system is a different UID than the MySQL user in the container:

id mysql
uid=112(mysql) gid=118(mysql) groups=118(mysql)

When I do a top -u 107 command, I get the output below which is basically the exact same services being run as it was when it was the landscape user (and remember above that the UID for landscape was also 107):

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                 
  58166 107       20   0  119160  29928   3668 S   7.3   0.4   9:45.91 carbon-cache                                            
  58266 107       20   0  400292   2572   1760 S   0.7   0.0   2:35.07 uwsgi                                                   
  58356 107       20   0 2093932 587804  16052 S   0.7   7.2  17:28.52 mysqld                                                  
  55356 107       20   0   76192   2700   1780 S   0.0   0.0   0:04.05 uwsgi                                                   
  55431 107       20   0  559180   2960   1972 S   0.0   0.0   0:20.28 turnserver                                              
  58125 107       20   0    4380    768    644 S   0.0   0.0   0:01.45 anvil                                                   
  58129 107       20   0    5048   2148   1532 S   0.0   0.0   0:02.41 stats                                                   
  58131 107       20   0  754016   2992   2992 S   0.0   0.0   0:00.34 node                                                    
  58138 107       20   0  684644  11556   7104 S   0.0   0.1   0:00.26 node                                                    
  58139 107       20   0  148136   6424   5836 S   0.0   0.1   0:00.74 postmaster                                              
  58158 107       20   0   23124   2820   2432 S   0.0   0.0   0:02.68 auth                                                    
  58159 107       20   0  148276   6356   5700 S   0.0   0.1   0:00.13 postmaster                                              
  58160 107       20   0  148136   4356   3800 S   0.0   0.1   0:00.57 postmaster                                              
  58161 107       20   0  148136   3740   3208 S   0.0   0.0   0:00.87 postmaster                                              
  58162 107       20   0  148832   3812   3096 S   0.0   0.0   0:00.46 postmaster                                              
  58163 107       20   0   67336   1428    740 S   0.0   0.0   0:01.03 postmaster                                              
  58164 107       20   0  148672   1988   1380 S   0.0   0.0   0:00.02 postmaster                                              
  58184 107       20   0 1845196  37176   8872 S   0.0   0.5   3:19.42 mongod                                                  
  58188 107       20   0  607972  39000   8752 S   0.0   0.5   0:05.85 node                                                    
  58265 107       20   0  388688   2500   1760 S   0.0   0.0   0:00.28 uwsgi                                                   
  97998 107       20   0  152832  18840  15760 S   0.0   0.2   0:00.26 postmaster                                              
 725610 107       20   0  149768   9380   8932 S   0.0   0.1   0:00.00 postmaster                                              
1353755 107       20   0  150720  20448  17460 S   0.0   0.3   0:00.01 postmaster                                              
1369625 107       20   0  150432  19640  16944 S   0.0   0.2   0:00.01 postmaster  

Notice the above services are all what Cloudron would be running too (I think anyways), like node, mysql, mongodb, turnserver, etc.

When I run top | grep mysql I get both UID 107 running it alongside the actual mysql user too:

top | grep mysql
  58356 107       20   0 2092908 587516  16008 S   0.8   7.2  17:33.18 mysqld                                                  
    638 mysql     20   0 2094924 124588  10788 S   0.0   1.5   5:29.17 mysqld       

In other words, I sort of feel like I didn't really gain any performance or reduce memory usage by removing landscape. Maybe I'm looking at it wrong though?

Does the above make any sense? I'm not quite following how this works. It almost seems like the landscape-common was uninstalled from the system which removed the landscape user too, but it's effectively still running just as a UID instead of any username. But am I totally misunderstanding this? lol Any insight would be awesome!

PS - I guess this isn’t a Cloudron item so much as an Ubuntu thing, so maybe this should be pushed to Discussion rather than Support? Totally fine with me if you’d prefer that.

I was checking in on my emails today that I had setup the other day as mailboxes with sieve rules to try and filter out spam. However it doesn’t actually appear to be working; which is odd because I tested them myself and they seemed correct. I’m going to make a small tweak and test again. Originally I had “Yes” and then I changed it to “Yes,” to remove false-positives in case any of the rules contained “yes” in their character set, but maybe that comma is throwing it off.

Update: I removed landscape tonight. I had to run a different command because the one provided and also seen in many other articles online couldn't find two of the packages so threw this error:

sudo apt-get purge landscape-client landscape-client-ui landscape-client-ui-install landscape-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package landscape-client-ui
E: Unable to locate package landscape-client-ui-install

So in order for it to work, I had to run this command instead:

sudo apt-get purge landscape-client landscape-common
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Package 'landscape-client' is not installed, so not removed
The following packages will be REMOVED:
  landscape-common*
0 upgraded, 0 newly installed, 1 to remove and 20 not upgraded.
After this operation, 410 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 112984 files and directories currently installed.)
Removing landscape-common (19.12-0ubuntu4.2) ...
Processing triggers for man-db (2.9.1-1) ...
(Reading database ... 112904 files and directories currently installed.)
Purging configuration files for landscape-common (19.12-0ubuntu4.2) ...

What's weird though is that after removing landscape, and rebooting the server, I still see the processes running as the same user ID (107), so I'm not sure if I really solved anything

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                               
  15631 yellowt+  20   0  413968 257640  22380 S   0.0   3.2   0:05.81 ruby2.7                                                                               
  15700 yellowt+  20   0 1144740 256476  11404 S   0.0   3.1   0:00.13 ruby2.7                                                                               
  15630 yellowt+  20   0  468764 254692  22740 S   0.0   3.1   0:05.35 ruby2.7                                                                               
  15701 yellowt+  20   0 1144740 251736   9608 S   0.0   3.1   0:00.04 ruby2.7                                                                               
  12150 107       20   0 2091216 240492  37868 S  28.5   2.9   0:05.46 mysqld     

@girish - I agree, it's likely some abstraction layer of some sort rather than it actually running MySQL, though what's odd is the memory usage is higher for the landscape lines than the yellowtent or mysql ones.

Here's what I can see, if this helps. It seems it duplicates almost everything that Cloudron would be running such as MySQL, PostegreSQL, MongoDB, Dovecot, Turn, etc.

id landscape
uid=107(landscape) gid=113(landscape) groups=113(landscape)

ps aux | head -1; ps aux | sort -rnk 4 | grep land*
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
landsca+  944441  1.9  6.7 2114148 554240 ?      Sl   May26  59:12 /usr/sbin/mysqld
landsca+  944442  0.0  0.5 617712 42208 ?        Sl   May26   0:22 node /app/code/service.js
landsca+  945567  0.3  0.3 1907948 30932 ?       Sl   May26  10:30 mongod --config /etc/mongodb.conf --auth
landsca+  942747  1.0  0.2 118748 22820 ?        Sl   May26  30:43 /usr/bin/python3 /usr/bin/carbon-cache --debug --config=/etc/carbon/carbon.conf start
landsca+  877995  0.0  0.2 150432 19188 ?        Ss   17:31   0:00 postgres: 12/main: usere18fa18bb1764b25bf784d82c022b0f5 dbe18fa18bb1764b25bf784d82c022b0f5 172.18.17.160(59512) idle
landsca+  869854  0.0  0.2 150432 18928 ?        Ss   17:25   0:00 postgres: 12/main: usere18fa18bb1764b25bf784d82c022b0f5 dbe18fa18bb1764b25bf784d82c022b0f5 172.18.17.160(53876) idle
landsca+  955217  0.0  0.1 156708 15912 ?        Ss   May26   0:00 postgres: 12/main: usere18fa18bb1764b25bf784d82c022b0f5 dbe18fa18bb1764b25bf784d82c022b0f5 172.18.17.160(36962) idle
landsca+  945568  0.0  0.0 887140  1648 ?        Sl   May26   0:01 node /app/code/service.js
landsca+  943437  0.0  0.0 394036  6564 ?        Sl   May26   0:01 uwsgi --master --workers 2 --buffer-size 16384 --no-orphans --ini /etc/uwsgi/apps-enabled/graphite-uwsgi.ini
landsca+  943435  0.0  0.0 395640  6276 ?        Sl   May26   0:00 uwsgi --master --workers 2 --buffer-size 16384 --no-orphans --ini /etc/uwsgi/apps-enabled/graphite-uwsgi.ini
landsca+  942745  0.0  0.0  76192    80 ?        S    May26   0:06 uwsgi --master --workers 2 --buffer-size 16384 --no-orphans --ini /etc/uwsgi/apps-enabled/graphite-uwsgi.ini
landsca+  940938  0.0  0.0 148672   812 ?        Ss   May26   0:00 postgres: 12/main: logical replication launcher   
landsca+  940937  0.0  0.0  67336   980 ?        Ss   May26   0:04 postgres: 12/main: stats collector   
landsca+  940936  0.0  0.0 148832  3268 ?        Ss   May26   0:01 postgres: 12/main: autovacuum launcher   
landsca+  940935  0.0  0.0 148136  1620 ?        Ss   May26   0:02 postgres: 12/main: walwriter   
landsca+  940934  0.0  0.0 148136  1644 ?        Ss   May26   0:01 postgres: 12/main: background writer   
landsca+  940932  0.0  0.0 148292  6108 ?        Ss   May26   0:00 postgres: 12/main: checkpointer   
landsca+  940656  0.0  0.0 883480  5988 ?        Sl   May26   0:00 node /app/code/service.js
landsca+  940655  0.0  0.0 148136  3584 ?        S    May26   0:03 /usr/lib/postgresql/12/bin/postmaster --config-file=/etc/postgresql/12/main/postgresql.conf
landsca+  934057  0.0  0.0 559180   956 ?        Ssl  May26   1:19 /usr/bin/turnserver -c /run/turnserver/turnserver.conf --pidfile /run/turnserver/turnserver.pid
landsca+  676796  0.0  0.0 149268  7528 ?        Ss   15:04   0:00 postgres: 12/main: root postgres 127.0.0.1(49710) idle
landsca+ 3876135  0.0  0.0 149268  4984 ?        Ss   03:04   0:00 postgres: 12/main: root postgres 127.0.0.1(52774) idle
landsca+ 3505163  0.0  0.0  23128  1720 ?        S    May27   0:03 dovecot/auth
landsca+ 3505162  0.0  0.0   5040  1680 ?        S    May27   0:03 dovecot/stats
landsca+ 3505117  0.0  0.0   4380   148 ?        S    May27   0:01 dovecot/anvil
landsca+ 2867491  0.0  0.0 149268  2816 ?        Ss   May27   0:00 postgres: 12/main: root postgres 127.0.0.1(47314) idle
landsca+ 1867396  0.0  0.0 149268  2552 ?        Ss   May27   0:00 postgres: 12/main: root postgres 127.0.0.1(51906) idle

Here's an example of the mysql memory differences (basically 554 MB used by the landscape version, and just over 100 MB by the mysql user which I assume is the one used by Cloudron?):

ps aux | head -1; ps aux | sort -rnk 4 | grep mysql*
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
landsca+  944441  1.9  6.7 2115172 554368 ?      Sl   May26  59:14 /usr/sbin/mysqld
mysql      57251  0.5  1.2 2099864 104272 ?      Ssl  May17  93:44 /usr/sbin/mysqld

@nebulon Excellent, thanks! I'll try to remove those later tonight after taking a VM snapshot to be extra safe. I've never seen or noticed "landscape" user before so I guess that was already absent from previous web hosts implementations of Ubuntu.

Just read about Vultr launching a new "Marketplace", thinking Cloudron may want to take advantage of that since it has 1-click image deployments for a few other platforms like AWS and DigitalOcean.

https://www.vultr.com/marketplace/become-a-verified-vendor/

https://www.vultr.com/news/Introducing-Vultr-Marketplace-One-Click-Setup-for-Popular-Apps-Stacks/

I was just looking at top output, and sorted by memory usage. The top user is mysqld which makes sense, but the user for it is a user named landscape (had to look it up in the /etc/passwd file). I don't believe I ever saw that user before.

I'm on a new Vultr server, so I suspect this may be part of their Ubuntu image by default. Any insight into whether I can remove this user / mysqld process from running with landscape user account? I did a quick search and seems landscape may be referring to https://landscape.canonical.com

Trying to make sure that my server isn't using memory unnecessarily. If that process doesn't need to be running, then I'd like to kill it and prevent it from running. I noticed there is a mysqld process run by the mysql user which I assume is the one Cloudron uses?

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                      
 944441 landsca+  20   0 2117548 512052  12652 S   0.7   6.3  46:07.32 mysqld                       
 948075 yellowt+  20   0 1212232 197468   7428 S   0.0   2.4   1:03.32 ruby2.7                      
 948080 yellowt+  20   0 1212232 165064   7732 S   0.0   2.0   1:04.72 ruby2.7                      
 947549 yellowt+  20   0  411812 130972   3468 S   0.0   1.6   2:18.52 ruby2.7                      
  57251 mysql     20   0 2099864 111712   4924 S   0.0   1.4  89:31.31 mysqld                       
    762 yellowt+  20   0  983452 110204  16368 S   0.0   1.4 286:15.49 node                         
 948338 yellowt+  20   0   10.9g 100668  14156 S   0.0   1.2   3:26.48 node                         
 947547 yellowt+  20   0  468668  99380   1984 S   0.0   1.2   0:21.64 ruby2.7                      
4099406 www-data  20   0  162064  80756  34464 S   0.0   1.0   0:00.26 php                          
3505159 yellowt+  20   0  111756  77764   6232 S   0.0   1.0   0:17.31 spamd child                  
4101518 www-data  20   0  157532  76676  34256 S   0.0   0.9   0:00.21 php7.4                       
4089812 www-data  20   0  386708  76260  55560 S   0.0   0.9   0:00.80 /usr/sbin/apach              
4099413 www-data  20   0  157568  76092  34260 S   0.0   0.9   0:00.22 php                          
4101528 www-data  20   0  157568  75896  34060 S   0.0   0.9   0:00.21 php7.4                       
4088551 www-data  20   0  380776  74264  59220 S   0.0   0.9   0:00.54 /usr/sbin/apach               

If it's the Ubuntu Landscape subsystems that I'm thinking it is, I found https://blog.giantgeek.com/?p=1409 which speaks to how to remove it. Anyone have experience with doing this, is it "safe"?

@robi said in Ability for spam messages to not be forward via mailing list:

@d19dotca great, now just document how it's applied and enabled for everyone else.

Edited with steps now

I performed a test, and believe the following sieve rules will work for anyone who wants to achieve this using a mailbox instead of mailing list.

# rule:[Discard spam messages on arrival]
if allof (header :contains "x-spam-status" "Yes,")
{
	discard;
	stop;
}
# rule:[Forward non-spam messages]
if allof (not header :contains "x-spam-status" "Yes,")
{
	redirect "<externalEmailAddress>";
}

For future visitors... To use the sieve filter above:

1. Login to Roundcube webmail app
2. Go to Settings > Filters
3. Click Actions > Edit Filter Set
4. Paste the above snippet, being sure to replace <externalEmailAddress> with the actual recipient's external email address
5. Save

Ideally though, I think there still needs to be a filter to discard/delete all spam messages sent to a mailing list, as spam should never arrive on a mailing list (regardless if it's used simply for forwarding or for an actual mailing list).

@girish Yes, it was more for a matter of maintenance that I chose to use the Mailing List functionality instead of a full mailbox. I suppose that's a workaround (though sounds like forwarding wasn't even working anyways until fixed in 6.3?), it's just not as "clean" or "convenient" to do. I think most people would agree that spam should never be sent to a mailing list (or at least not forwarded on from the mailing list), so I'd hope for a cleaner less-overhead manner to prevent spam from being forwarded to mailing lists. Hopefully that makes sense, but let me know if I can clarify at all.

@girish said in What's coming in Cloudron 6.3:

BTW, about the spam fix, it's easy to check if an email is marked as spam and drop it (i.e based on spamassassin flags). However, since we don't have a place to quarantine yet, you will only see a eventlog entry that it's gone. Is this OK for the moment till we implement a monitorable mail queue?

I think that's fine for now, but would it be possible to only affect mailing lists perhaps (assuming this change is mostly just temporary and probably will only really be used by me for the time being). I ask because I think if I were to just drop all messages marked as spam from all mail delivery completely, it'd be a detriment to my hosted mailboxes. The only place I want to disable sending spam mail is for mailing lists alone, if possible. But either way I guess, better than nothing I need to ensure Gmail isn't going to block mail from my server completely just because the two addresses my mailing lists forward to are total magnets for spam.

@girish Oh sure, that'd be good. So there's going to be a bug fix version then I presume with what's already been done so far (such as 6.2.9 maybe or still 6.3.0)? And then email + security will be added to something like 6.4 instead if the next release is still 6.3?

@dylightful I think it's a nice idea to add reCAPTCHA / hCaptcha as needed to the page. With that said, as @marcusquinn stated, fail2ban should more or less prevent any brute force attacks. Also the Cloudron has rate limits in place by default (https://docs.cloudron.io/security/#rate-limits) for Cloudron login page. Of course, that can be greatly improved as 10 requests per second per IP is far too high in my opinion, should be more like 10 requests per 5 or 10 minutes or something like that. But that was also requested already too to improve the rate limits to be more secure: https://forum.cloudron.io/post/28271 which @girish has already confirmed is going to be one of the focuses in 6.3.

@girish Please tell me 6.3 is coming down the pipe this week. I'm so eager for it! Desperately needing some of these email improvements, specifically the most urgent for me is the ability to not forward email on to mailing lists if it's been identified as spam. The limitations currently are impacting the trust of my mail server by other systems like Gmail which is rate limiting my emails now because so much spam is forwarded on to a couple of Gmail addresses via the mailing list functionality. At least they're not outright blocking me, but that'd be the next logical step that I want to avoid!

@girish said in Roundcube - Package updates:

[2.5.1]

• Add separate forwarding and vacation section

Is it possible to expand on the "add separate forwarding and vacation section". I have a lot of users using Roundcube and just want to be sure I know what to expect or notify them of if needed, as many have various rules set for auto-responses and such. It doesn't seem this was an update to Roundcube based on their GitHub releases, so curious what was changed.

I found https://git.cloudron.io/cloudron/roundcube-app/-/commit/b8b2caac3c54d08d4e14e2d7cafc62224a8a4451 - this looks like it's more platform-related then, no actual UI changes for users, right?

@girish Always so quick to fix these things - thank you so much for your attention to this as always!

@girish Thank you for clarifying Girish. Yes please, this should be added to Cloudron. I'd assume any user would expect all settings to be restored when restoring from a backup other than any data that can cause conflicts with existing instances (i.e. redirect hostnames that already exist, etc) or is just generally unnecessary (though I guess that's subjective) such as everything in the Graphs tab. I'd think everything else that won't cause conflicts should be restored using an app backup - including app icons.

I just realized when cloning an app from backup, the icon that was assigned to an app is not transferred to the new instance of it when using the Clone From This Backup function. I presume this may also be the case if restoring in-place, but have not tested that task specifically.

To reproduce:

1. Add custom icon to any app, we'll refer to this app as "App #1"
2. Make a backup of App #1
3. View the Backup tab on App #1 and select the Clone From This Backup button to clone to a new app URL
4. The new app (App #2) is using the default icon rather than the custom one set in step 1 above

I assume the backups may not have been designed with this use-case in mind so it may not technically be a "defect" but I suspect this is a gap in user experience as I believe any user would expect their custom app icons to be included in the app backups and thus restored when cloning or restoring from a backup too.

Please make sure the custom app icons are included in the app backups. I spent all that time customizing my app icons, so it's a bit annoying whenever I'm cloning an app and it gets the old icon instead.

@privsec Note, just because it's "running" doesn't mean it's running well.

@girish Perfect, looking forward to 6.3. I desperately need the ability to block messages identified as spam from being forwarded on to external addresses by way of the mailing lists feature. Gmail is constantly rate limiting mail from my server (which isn't a big deal in the scheme of things, better than being outright block listed, lol) because of how much spam goes to one specific address which forwards to their preferred address on Gmail. Would love to get rid of this behaviour by basically just dropping spam messages to external addresses if the new feature is enabled to do so.

On a note more relevant to this topic though... what was the root cause of this very strange behaviour? I'm most curious, haha.

@artd2019 Currently Cloudron does not offer (a documented) "whitelist" / "allow list" function, so there is no way to guarantee deliverability of an incoming email. There is a feature request for it already that I'd encourage you to upvote. Version 6.3 coming in the not too distant future is supposed to deliver on many of the email improvements users have been clamouring for (including myself).

One thing you can do if you're not already is making sure to remove those emails from the Junk / Spam folder in order for SpamAssassin to unlearn the message as spam. Eventually after a sufficient number of them, it'll learn that those emails are not spam and will decrease the BAYES probability of it being spam.

On a related topic, you may want to consider enhancing your SpamAssassin rules list from the defaults, specifically in your case the BAYES_## rules perhaps as I have offered previously in this forum.