Topics created by d19dotca

Just encountered the issue again. 😞

Here are the latest logs:

Jun 17 03:13:17 my kernel: redis-server invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 Jun 17 03:13:17 my kernel: CPU: 1 PID: 6436 Comm: redis-server Not tainted 5.4.0-74-generic #83-Ubuntu Jun 17 03:13:17 my kernel: Hardware name: Vultr HFC, BIOS Jun 17 03:13:17 my kernel: Call Trace: Jun 17 03:13:17 my kernel: dump_stack+0x6d/0x8b Jun 17 03:13:17 my kernel: dump_header+0x4f/0x1eb Jun 17 03:13:17 my kernel: oom_kill_process.cold+0xb/0x10 Jun 17 03:13:17 my kernel: out_of_memory.part.0+0x1df/0x3d0 Jun 17 03:13:17 my kernel: out_of_memory+0x6d/0xd0 Jun 17 03:13:17 my kernel: __alloc_pages_slowpath+0xd5e/0xe50 Jun 17 03:13:17 my kernel: __alloc_pages_nodemask+0x2d0/0x320 Jun 17 03:13:17 my kernel: alloc_pages_current+0x87/0xe0 Jun 17 03:13:17 my kernel: __page_cache_alloc+0x72/0x90 Jun 17 03:13:17 my kernel: pagecache_get_page+0xbf/0x300 Jun 17 03:13:17 my kernel: filemap_fault+0x6b2/0xa50 Jun 17 03:13:17 my kernel: ? unlock_page_memcg+0x12/0x20 Jun 17 03:13:17 my kernel: ? page_add_file_rmap+0xff/0x1a0 Jun 17 03:13:17 my kernel: ? xas_load+0xd/0x80 Jun 17 03:13:17 my kernel: ? xas_find+0x17f/0x1c0 Jun 17 03:13:17 my kernel: ? filemap_map_pages+0x24c/0x380 Jun 17 03:13:17 my kernel: ext4_filemap_fault+0x32/0x50 Jun 17 03:13:17 my kernel: __do_fault+0x3c/0x130 Jun 17 03:13:17 my kernel: do_fault+0x24b/0x640 Jun 17 03:13:17 my kernel: __handle_mm_fault+0x4c5/0x7a0 Jun 17 03:13:17 my kernel: handle_mm_fault+0xca/0x200 Jun 17 03:13:17 my kernel: do_user_addr_fault+0x1f9/0x450 Jun 17 03:13:17 my kernel: __do_page_fault+0x58/0x90 Jun 17 03:13:17 my kernel: do_page_fault+0x2c/0xe0 Jun 17 03:13:17 my kernel: do_async_page_fault+0x39/0x70 Jun 17 03:13:17 my kernel: async_page_fault+0x34/0x40 Jun 17 03:13:17 my kernel: RIP: 0033:0x5642ded5def4 Jun 17 03:13:17 my kernel: Code: Bad RIP value. Jun 17 03:13:17 my kernel: RSP: 002b:00007ffe39d1f480 EFLAGS: 00010293 Jun 17 03:13:17 my kernel: RAX: 0000000000000001 RBX: 0000000000000000 RCX: 00005642dedb5a14 Jun 17 03:13:17 my kernel: RDX: 0000000000000001 RSI: 00007f6c13408823 RDI: 0000000000000002 Jun 17 03:13:17 my kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: 00007ffe39d1f3d8 Jun 17 03:13:17 my kernel: R10: 00007ffe39d1f3d0 R11: 0000000000000000 R12: 00007ffe39d1f648 Jun 17 03:13:17 my kernel: R13: 0000000000000002 R14: 00007f6c1341f020 R15: 0000000000000000 Jun 17 03:13:17 my kernel: Mem-Info: Jun 17 03:13:17 my kernel: active_anon:961079 inactive_anon:871527 isolated_anon:0 Jun 17 03:13:17 my kernel: Node 0 active_anon:3844316kB inactive_anon:3486108kB active_file:624kB inactive_file:464kB unevictable:18540kB isolated(anon):0kB isolated(file):20kB mapped:900656kB dirty:0kB writeback:0kB shmem:1207700kB shmem_thp: 0kB shmem_pmdmapped: 0kB anon_thp: 0kB writeback_tmp:0kB unstable:0kB all_unreclaimable? no Jun 17 03:13:17 my kernel: Node 0 DMA free:15900kB min:132kB low:164kB high:196kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15992kB managed:15908kB mlocked:0kB kernel_stack:0kB pagetables:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB Jun 17 03:13:17 my kernel: lowmem_reserve[]: 0 2911 7858 7858 7858 Jun 17 03:13:17 my kernel: Node 0 DMA32 free:44284kB min:24988kB low:31232kB high:37476kB active_anon:1420788kB inactive_anon:1398328kB active_file:0kB inactive_file:360kB unevictable:0kB writepending:0kB present:3129200kB managed:3063664kB mlocked:0kB kernel_stack:9504kB pagetables:29712kB bounce:0kB free_pcp:1560kB local_pcp:540kB free_cma:0kB Jun 17 03:13:17 my kernel: lowmem_reserve[]: 0 0 4946 4946 4946 Jun 17 03:13:17 my kernel: Node 0 Normal free:42004kB min:42460kB low:53072kB high:63684kB active_anon:2423528kB inactive_anon:2087780kB active_file:588kB inactive_file:808kB unevictable:18540kB writepending:0kB present:5242880kB managed:5073392kB mlocked:18540kB kernel_stack:29040kB pagetables:53096kB bounce:0kB free_pcp:3512kB local_pcp:676kB free_cma:0kB Jun 17 03:13:17 my kernel: lowmem_reserve[]: 0 0 0 0 0 Jun 17 03:13:17 my kernel: Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB Jun 17 03:13:17 my kernel: Node 0 DMA32: 3172*4kB (UME) 1384*8kB (UE) 744*16kB (UME) 242*32kB (UME) 15*64kB (UME) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 44368kB Jun 17 03:13:17 my kernel: Node 0 Normal: 157*4kB (UME) 2012*8kB (UMEH) 976*16kB (UMEH) 291*32kB (UME) 4*64kB (M) 2*128kB (M) 1*256kB (M) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 42420kB Jun 17 03:13:17 my kernel: Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Jun 17 03:13:17 my kernel: 337832 total pagecache pages Jun 17 03:13:17 my kernel: 33532 pages in swap cache Jun 17 03:13:17 my kernel: Swap cache stats: add 413059, delete 379528, find 3909879/3976154 Jun 17 03:13:17 my kernel: Free swap = 0kB Jun 17 03:13:17 my kernel: Total swap = 1004540kB Jun 17 03:13:17 my kernel: 2097018 pages RAM Jun 17 03:13:17 my kernel: 0 pages HighMem/MovableOnly Jun 17 03:13:17 my kernel: 58777 pages reserved Jun 17 03:13:17 my kernel: 0 pages cma reserved Jun 17 03:13:17 my kernel: 0 pages hwpoisoned [...] Jun 17 03:13:17 my kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=ac3d3b89315a33f079b879cebf2fa7d68e0512640f9a7ab16e1b9c79b192fa1b,mems_allowed=0,global_oom,task_memcg=/system.slice/box-task-10731.service,task=node,pid=1757378,uid=0 Jun 17 03:13:17 my kernel: Out of memory: Killed process 1757378 (node) total-vm:2615204kB, anon-rss:1725444kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8244kB oom_score_adj:0 Jun 17 03:13:17 my kernel: oom_reaper: reaped process 1757378 (node), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB Jun 17 03:13:17 my sudo[1757377]: pam_unix(sudo:session): session closed for user root Jun 17 03:13:17 my systemd[1]: box-task-10731.service: Main process exited, code=exited, status=50/n/a Jun 17 03:13:17 my systemd[1]: box-task-10731.service: Failed with result 'exit-code'.

This time it wasn't MySQL but was redis apparently?

The system memory output is below too:

free -h total used free shared buff/cache available Mem: 7.8Gi 4.5Gi 237Mi 1.2Gi 3.1Gi 1.8Gi Swap: 980Mi 969Mi 11Mi

Okay weird update... I changed it to 1G but still received the same error after restarting MySQL service. So then I went back and set it back to 64M default it had (which should have been enough anyways), but then when I tried it again soon after it then suddenly worked importing the JSON file without "Malformed communication packet", it's as if maybe the 1G value worked but wasn't recognized until later? Or it was a total coincidence, I'm not certain which yet. lol.

I'm wondering now if it maybe makes sense to up the max_allowed_packet up a bit though? I mean 64 MB is fine but wondering if it should be closer to 256 MB or something like that instead? I'm not certain if this was even to do with it, but it seemed possible given that it worked soon after tinkering with the values.

@d19dotca right..

otherwise it might be time to look through the mail container

@d19dotca said in Backups failing on same app each time:

By the way, since there's a ton of backup crashing that seem memory related... is it possible to improve the logging so that there's something in there saying "OutOfMemory" or something like that so poeple know why it's crashing?

Yeah, I would love this as well. Sadly, there seems to be no way on linux to know this reliably. A process just gets killed unceremoniously with SIGKILL and that's that 😕 If anyone know how this can be detected would be great.

We are live - https://www.vultr.com/marketplace/apps/cloudron

@girish - I stumbled across this article: https://blog.dbi-services.com/how-uid-mapping-works-in-docker-containers/

The article seems to imply (unless I'm misunderstanding it) that it may be a good practice to create a user on the host system that matches up with the user running the service in each container. So in other words, taking mysql as the example... on the Ubuntu host the mysql user should be a known user that's created with UID 2000 for example, and then in the Docker container for MySQL it'll also run with a UID of 2000. Then for something like mongodb, it'll be a host user with UID 2001 for example and then the user running mongodb in the container will also run with the UID of 2001. You're definitely allowed to create users with a specific UID, so hopefully this is all doable.

The above makes it a bit more secure from what I'm reading, plus of course for the "OCD" in all of us it keeps things cleaner and easier to understand what's happening when looking at top of a ps output for example.

Another article that is similar in nature is this one: https://medium.com/@mccode/understanding-how-uid-and-gid-work-in-docker-containers-c37a01d01cf

I'm just wondering if this is an improvement that should and could be made to Cloudron's images for at least the fundamental services that Cloudron deploys and runs in both host and containers.

I guess I'm not convinced the current setup is the ideal setup for people. Since Cloudron is mean to to be deployed on brand new Ubuntu systems, there should be no real need to accomodate other UIDs which may be present on some providers and not others because you can simply choose a really high UID that no default Ubuntu image should be using. Hopefully the above makes sense. Please correct me if I'm misunderstanding some of this.

@girish Always so quick to fix these things - thank you so much for your attention to this as always! 🙂

@d19dotca said in Loading Event Log page throws ER_OUT_OF_SORTMEMORY:

Basically the real settings are coming from /etc/mysql/mysql.cnf, right?

Yes, they are the same file as you found out. In any case, the change I made is good because ORDER BY <xx> should always have xx indexed in the database to lower memory use.

@d19dotca ah, noop is for the DNS. For the certs, if you click on advanced (in the Domains view), it will let you choose self-signed/fallback certs. That way it won't use let's encrypt.

@jdaviescoates Awesome, thanks for the tip! I just replaced the LAMP apps with Surfer for the redirect role, that saves some resources off MySQL and Redis which is nice. Thanks again. 🙂

Added in 6.3

@girish Correct. In my case they're almost all managed by Cloudron with some exceptions like extra records needed by Mailjet for example, but yeah the vast majority of the DNS records between all the domains I manage are Cloudron-related.

@d19dotca Swift based object stores are one of the least advanced.

IBMs is one of the most. (20GB free)

@d19dotca Perhaps even a per-app option, as in "Check for spam sent from this app."

@girish Seems the latest app package update fixes it. Thanks for the quick turnaround as always! 🙂

@d19dotca No problem for tagging me 🙂 I will take a look at this along with other mail related stuff before the next release (i.e so that any required fix is in the coming release).

I think the WP distribution does not have .htaccess files at all and it's only generated when you setup permalinks. I thought maybe atleast wp-content might have .htaccess inside it but even that doesn't. I guess it's up to the user to set those up, if they want to.

I guess the main danger is if your site somehow allows someone to upload php files, then it might get executed under wp-content. But this of course requires one to install such a plugin that let's someone upload random files.

@girish WOW that greatly simplifies the process, that's great, still always amazed at how CR makes sysadmin soooo much easier 🙂

I like both the suggestions! Will look into adding them.

@girish Thanks. Sendding now.

@nebulon Okay, that's fair. Yeah I tried in the Radicale docs but couldn't find anything really related to theming or any configurations for overriding the page title and stuff for the web UI. Seems strange, but I guess it's mostly used by admins rather than regular users so I guess it's not a biggie. Just been on a bit of a branding project slowly over the last while and Radicale seemed like it should be an easy one, but didn't turn out that way. haha. I'll dig a little more and file a report upstream for the feature if it truly doesn't exist yet. Thanks for the guidance. 🙂

PS - If it's not actually a feature though (doesn't seem like it so far), then this is another reason we should have more than one CalDAV/CardDAV server option 😉 If I get time over the next few weeks, maybe I"ll try to package one of them up as a sort of first package attempt.

@d19dotca thanks for reporting. The 404s only come from map files for the minified 3rdparty libraries here. Those are only fetched if you open the webinspector to enabled nicer debugging, so those are non-fatal and can safely be ignored.

@girish said in Anyone else see many connections denied due to "Mail from domain <domain> is not allowed from your host" repeatedly from spammy IPs?:

A bit of a wild guess: mail from is usually <> for bounce mail. So, this seems like some poor denial of service or maybe those IPs know that some mail software misbehaves with such carefully crafted mail.

Ah very interesting, I appreciate that insight. It was definitely strange when I saw it happening - so many requests at once. I'll keep an eye out for it. Sounds like it's all good then as far as Cloudron is concerned. 🙂 Thanks Girish.

@d19dotca it's not a problem. Just the natue of any service being exposed to the internet. There a bots, misconfigured services, compromised iot devices doing all sorts of things. Nothing to worry.