Cloudron Forum

@nebulon I’m wondering… since Cloudron depends entirely on Docker networking to function… is there maybe room to improve the IP blocklist checking so that it ignores any entries of the current Docker networking ranges such as 172.18.xxx.xxx addresses? It feels to me like there would never be a use-case to block those, and while we certainly need to use reliable IP lists (lesson learned, haha), I also wonder if this feature should be improved in the future to ignore any private IPs or at least any Docker IPs.

@sponch said in Sharing custom SpamAssassin Rules:

@d19dotca great and thanks! for abusix I just have to put in the api key without <>, right?
Done but don't get queries shown in the dashboard (though I sent some mails).
Using zen.spamhaus.org as DNSBL

That’s correct, no angle brackets. The full URL to use is shown in the Abusix dashboard but it’s really just the API key plus the subdomain parts.

I didn’t see queries until the following day I think, if I’m remembering correctly. So maybe give it another day or two? Also maybe make sure you don’t have any spaces or blank characters in the DNSBL just in case that’s throwing off the DNS queries to it.

Also I saw you mentioned that you didn’t see on the dashboard “though [you] sent some mails”… just to clarify, the queries will be done when you receive mail rather than send mail. I’m sure you knew that, but just in case, I thought I should clarify that part.

If you don’t see anything in a couple of days on the dashboard then let me know, and I can try to help. If it’s set correctly in Cloudron though then it could be something more on the Abusix side, maybe something needs to get confirmed or activated first (I don’t remember having to do that though but I’ve been using it for a while so I can’t remember the full on-boarding workflow).

Just FYI, the botscout_30 ipset is the one that contained the 172.18.0.2 and 172.18.0.4 addresses. So I guess the botscout_30d list cannot be fully trusted for now.

OMG I think I figured it out and I'm really kicking myself. I threw a bunch of logs from Cloudron into ChatGPT for review, and it highlighted a line in the logs:

2025-04-13T01:02:17.357Z box:shell network /usr/bin/sudo -S /home/yellowtent/box/src/scripts/setblocklist.sh

I believe this is from the ipblocklist workflow I implemented based on https://docs.cloudron.io/guides/community/blocklist-updates/. I took a look at the IP address concatenation and sure enough it includes the two IP addresses of 172.18.0.2 and 172.18.0.4, which represent the two redis containers currently failing. I think this explains why this is happening, so it was basically the result of these IPs somehow making it onto the blocklist from firehol lists. Going to determine which list specifically added it so I can remove it from my IP blocklists I guess. As soon as I cleared the IP blocklist in Cloudron, everything worked immediately.

Sorry for the blast of comments, but figured it may help others who run into similar issues. I'm just glad we sorted it out.

When I exec into the working container and run the health-check to the broken container, it succeeds. So this definitely confirms to me that the issue is somewhere from the host network to container, where-as container-to-container works perfectly fine.

From the working container to the non-working container:

root@redis-2e36b3e8-22d9-477c-bd56-d2c284909932:/app/code# curl -v http://172.18.0.2:3000/healthcheck
*   Trying 172.18.0.2:3000...
* Connected to 172.18.0.2 (172.18.0.2) port 3000
> GET /healthcheck HTTP/1.1
> Host: 172.18.0.2:3000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-vyGp6PvFo4RvsFtPoIWeCReyIC8"
< Date: Fri, 18 Apr 2025 05:24:05 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host 172.18.0.2 left intact

From the non-working container to the working container:

root@redis-00895422-a1ff-4196-8bb8-cb4ff8d6eeaa:/app/code# curl -v http://172.18.0.3:3000/healthcheck
*   Trying 172.18.0.3:3000...
* Connected to 172.18.0.3 (172.18.0.3) port 3000
> GET /healthcheck HTTP/1.1
> Host: 172.18.0.3:3000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-vyGp6PvFo4RvsFtPoIWeCReyIC8"
< Date: Fri, 18 Apr 2025 05:25:40 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host 172.18.0.3 left intact

But from host to working redis and non-working redis shows results above at https://forum.cloudron.io/post/105899

This seems like a port binding issue or Docker network issue somehow, because the health check GET requests work successfully for any working containers, but the non-working containers timeout. However when inside the non-working container, the health check (if using localhost) works fine, so redis is working and everything seems to be running successfully, the issue appears to be more that it can't be accessed for some reason from the host.

@nebulon / @girish , is there a possibility that this is related to https://forum.cloudron.io/post/103522 at all? I wouldn't think so since this seems to be intermittent and only affecting one container at a time (usually redis it seems)... but wanted to make sure you were aware just in case this is the root cause.

@nebulon Okay I see the following when I run that healthcheck GET with the current yellow light causing issues again which is blocking the backup process. It does indeed seem like a possible connection issue:

root@my:~# curl -v http://172.18.0.4:3000/healthcheck
*   Trying 172.18.0.4:3000...
* connect to 172.18.0.4 port 3000 from 172.18.0.1 port 44402 failed: Connection timed out
* Failed to connect to 172.18.0.4 port 3000 after 135822 ms: Couldn't connect to server
* Closing connection
curl: (28) Failed to connect to 172.18.0.4 port 3000 after 135822 ms: Couldn't connect to server

When I use a working redis instance showing the green lights, then I get the expected response:

root@my:~# curl -v http://172.18.0.7:3000/healthcheck
*   Trying 172.18.0.7:3000...
* Connected to 172.18.0.7 (172.18.0.7) port 3000
> GET /healthcheck HTTP/1.1
> Host: 172.18.0.7:3000
> User-Agent: curl/8.5.0
> Accept: */*
> 
< HTTP/1.1 401 Unauthorized
< X-Powered-By: Express
< Content-Type: application/json; charset=utf-8
< Content-Length: 2
< ETag: W/"2-vyGp6PvFo4RvsFtPoIWeCReyIC8"
< Date: Thu, 17 Apr 2025 18:43:37 GMT
< Connection: keep-alive
< Keep-Alive: timeout=5
< 
* Connection #0 to host 172.18.0.7 left intact

Any recommendations? I've never had the frequency of these issues before until the past month or two, and not sure why this is happening. Unclear if this is something in my environment (it seems like maybe others are not seeing this issue), or if it's a Cloudron thing. Certainly open to this maybe being an issue in my environment only, just not sure what to try next.

@Supaiku Is it the Developer edition? That happens for me too but I believe that’s intentional as the developer edition is meant to provide more control to the admins. With that said, I also have all auto-updates (within WordPress including core) disabled, so that may be part of it too (especially if that’s how you’ve configured it as well).

@humptydumpty Unfortunately there isn’t a way (that I’m aware of at least) to filter out Gmail spam from normal Gmail messages, as Gmail servers tend to be whitelisted. That’s where the freemail rules can play a part though, and also the Bayesian learning scores too. Eventually the Bayesian learning will realize the contents of the spammy messages from Gmail and I have the scores set in such a way that once it’s fairly confident in it from past user interactions, similar emails even from Gmail will go to spam where it belongs, keeping the non-spam Gmail in the inbox.

I’m sure there may be better ways to improve it, but that’s all I’ve found that works decently for now on the free email providers like Gmail.

For a DNSBL drop list at connection time, I use Abusix which has been reliable but definitely on the conservative side (as you’d want on the connection drop list part to avoid false-positives), that might help a bit overall if it’s an email pretending to be from Gmail but not actually sent via Gmail.

@crazybrad said in Sharing custom SpamAssassin Rules:

@d19dotca Many thanks for sharing this. I'd like to suggest another addition based on one nasty abuse I've seen: unsubscribe links that use http instead of https, hoping that browser security blocks/warnings will cause users not to follow through and unsubscribe. Anyone not using https for anything these days is not worthy of my time:) This puts them where they belong:

# Rule to detect unsubscribe links that do not use HTTPS
body UNSUB_LINK_HTTP /unsubscribe.*http:\/\//i
describe UNSUB_LINK_HTTP Unsubscribe link does not use HTTPS
score UNSUB_LINK_HTTP 10.0

Oh I like that! That’s a great idea!

Aww thank you for the compliment! Honestly I’ve never started a tip jar but that’s probably a good idea for me to start one.

@murgero said in Sharing custom SpamAssassin Rules:

@d19dotca does this just go into email -> Spam Filter -> Custom Spam Assassin Settings?

Yes, it goes right there. Basically from the Mail page > Spam filtering > Custom Spamassassin Rules box.

You can copy & paste the entire thing, but do note a few items just in case:

• You will need to likely remove the blocklist_from or welcomelist_from lines unless you have emails to place in those two sections already, I left those there just for an example.
• If you want to use the DNSBLs from Abusix then you'll need to use your own API key (it's free for under 5,000 queries per day averaged over 7 days, it seems to work great and I highly recommend it).

The rest though you can basically copy & paste directly. Of course YMMV as they say, but this list works pretty well for me, or at least is a noticeable improvement over the rule tweaks I was using last year.

Okay everyone... I've spent the past month really fine-tuning the SpamAssassin rules (especially after SpamAssassin was upgraded to 4.0 with Cloudron 8.3), and it seems to be working quite well for me. Spam filtering is never perfect, but it has improved a lot since the last time I took a look at the rules. The list is much longer now, and I added many more rules along with some additional lists for extra checks too.

If you want to use Abusix (which I've found to be very reliable), then you'll need a free API key, and can replace <API_KEY> in the list below with your own.

As always, your own mileage may vary, but this should be an effective SpamAssassin rule set to use. Good luck.

# ============================
# Allowlist / Blocklist Rules
# ============================

score USER_IN_BLOCKLIST 50.0
score USER_IN_ALL_SPAM_TO -50.0
score USER_IN_MORE_SPAM_TO -10.0
score USER_IN_WELCOMELIST -50.0

## Block specific recipients/senders
blocklist_from <EMAIL>

## Allowlist specific recipients/senders
welcomelist_from <EMAIL>


# ============================
# Bayesian Filtering (BAYES)
# ============================

bayes_auto_learn 1
bayes_auto_learn_threshold_nonspam -4.0
bayes_auto_learn_threshold_spam 12.0

score BAYES_00 -6.0
score BAYES_05 -4.0
score BAYES_20 1.5
score BAYES_40 2.5
score BAYES_50 3.25
score BAYES_60 3.75
score BAYES_80 4.25
score BAYES_95 5.5
score BAYES_99 7.0
score BAYES_999 7.5


# ============================
# DNS-based Blocklists (DNSBL)
# ============================

## DNS Blocklists
score RCVD_IN_BL_SPAMCOP_NET 2.5
score RCVD_IN_MSPIKE_BL 0.0
score RCVD_IN_MSPIKE_L2 1.0
score RCVD_IN_MSPIKE_L3 1.5
score RCVD_IN_MSPIKE_L4 3.5
score RCVD_IN_MSPIKE_L5 5.0
score RCVD_IN_MSPIKE_ZBI 4.0
score RCVD_IN_PBL 3.5
score RCVD_IN_SBL 5.0
score RCVD_IN_SBL_CSS 4.0
score RCVD_IN_VALIDITY_RPBL 1.5
score RCVD_IN_XBL 6.0
score RCVD_IN_ZEN_BLOCKED 0.0
score RCVD_IN_ZEN_BLOCKED_OPENDNS 0.0

## DNS Whitelists
score RCVD_IN_DNSWL_BLOCKED 0.0
score RCVD_IN_DNSWL_HI -6.0
score RCVD_IN_DNSWL_LOW -2.5
score RCVD_IN_DNSWL_MED -4.5
score RCVD_IN_DNSWL_NONE 0.0
score RCVD_IN_MSPIKE_H2 -1.0
score RCVD_IN_MSPIKE_H3 -1.5
score RCVD_IN_MSPIKE_H4 -3.5
score RCVD_IN_MSPIKE_H5 -5.0
score RCVD_IN_MSPIKE_WL 0.0


# ============================
# URI Blocklists (URIBL)
# ============================

score URIBL_ABUSE_SURBL 6.0
score URIBL_BLACK 7.0
score URIBL_CR_SURBL 3.5
score URIBL_CSS 3.0
score URIBL_CSS_A 3.0
score URIBL_DBL_ABUSE_BOTCC 5.5
score URIBL_DBL_ABUSE_MALW 5.5
score URIBL_DBL_ABUSE_PHISH 5.5
score URIBL_DBL_ABUSE_REDIR 2.0
score URIBL_DBL_ABUSE_SPAM 5.5
score URIBL_DBL_BLOCKED 0.0
score URIBL_DBL_BLOCKED_OPENDNS 0.0
score URIBL_DBL_BOTNETCC 5.5
score URIBL_DBL_ERROR 0.0
score URIBL_DBL_MALWARE 5.5
score URIBL_DBL_PHISH 5.5
score URIBL_DBL_SPAM 5.5
score URIBL_GREY 0.5
score URIBL_MW_SURBL 5.0
score URIBL_PH_SURBL 5.0
score URIBL_RED 2.0
score URIBL_RHS_DOB 2.0
score URIBL_SBL 4.0
score URIBL_SBL_A 3.0
score URIBL_ZEN_BLOCKED 0.0
score URIBL_ZEN_BLOCKED_OPENDNS 0.0


# ============================
# Email Authentication (SPF/DKIM/ARC)
# ============================

## DKIM
score DKIM_INVALID 2.0
score DKIM_ADSP_ALL 2.0
score DKIM_ADSP_CUSTOM_MED 1.5
score DKIM_ADSP_NXDOMAIN 4.5
score DKIM_SIGNED 0.0
score DKIM_VALID 0.0
score DKIM_VALID_AU 0.0
score DKIM_VALID_EF 0.0
score DKIM_VERIFIED 0.0
score DKIMWL_BL 3.0
score DKIMWL_WL_HIGH -6.0
score DKIMWL_WL_MED -4.0
score DKIMWL_WL_MEDHI -4.5
score USER_IN_DEF_DKIM_WL -6.0

## SPF
score FORGED_SPF_HELO 4.0
score SPF_FAIL 4.0
score SPF_HELO_FAIL 2.5
score SPF_HELO_NEUTRAL 1.0
score SPF_HELO_NONE 0.25
score SPF_HELO_PASS -1.0
score SPF_HELO_SOFTFAIL 4.0
score SPF_NEUTRAL 0.0
score SPF_NONE 1.0
score SPF_PASS -1.0
score SPF_SOFTFAIL 1.5
score T_SPF_PERMERROR 2.0
score T_SPF_TEMPERROR 0.0
score USER_IN_DEF_SPF_WL -6.0

## ARC
score ARC_INVALID 2.0
score ARC_SIGNED 0.0
score ARC_VALID -0.5


# ============================
# HTML & MIME Structure Rules
# ============================

score HTML_FONT_LOW_CONTRAST 0.0
score HTML_FONT_SIZE_LARGE 2.0
score HTML_FONT_TINY_NORDNS 0.0
score HTML_IMAGE_ONLY_04 1.25
score HTML_IMAGE_ONLY_08 1.25
score HTML_IMAGE_ONLY_12 1.25
score HTML_IMAGE_ONLY_16 1.25
score HTML_IMAGE_ONLY_20 1.25
score HTML_IMAGE_ONLY_24 1.25
score HTML_IMAGE_ONLY_28 1.25
score HTML_IMAGE_ONLY_32 1.25
score HTML_MESSAGE 0.0
score HTML_MIME_NO_HTML_TAG 0.5
score HTML_OBFUSCATE_05_10 0.5
score HTML_OBFUSCATE_10_20 1.0
score HTML_OBFUSCATE_20_30 2.0
score HTML_OBFUSCATE_30_40 2.5
score HTML_OBFUSCATE_50_60 3.0
score HTML_OBFUSCATE_70_80 3.5
score HTML_OBFUSCATE_90_100 4.0
score HTML_SHORT_LINK_IMG_1 2.0
score HTML_SHORT_LINK_IMG_2 3.0
score HTML_SHORT_LINK_IMG_3 3.0
score MIME_HTML_MOSTLY 0.0
score MIME_HTML_ONLY 0.0
score MPART_ALT_DIFF 0.5
score MPART_ALT_DIFF_COUNT 0.5


# ============================
# Header / Envelope Heuristics
# ============================

score HEADER_FROM_DIFFERENT_DOMAINS 2.0
score MISSING_DATE 2.5
score MISSING_FROM 2.0
score MISSING_HEADERS 2.5
score MISSING_MID 1.0
score MISSING_SUBJECT 1.0
score MSGID_OUTLOOK_INVALID 4.0
score MISSING_HB_SEP 2.0
score TO_NO_BRKTS_FROM_MSSP 1.5
score REPLYTO_WITHOUT_TO_CC 2.5


# ============================
# Freemail & Identity Rules
# ============================

score FREEMAIL_ENVFROM_END_DIGIT 1.0
score FREEMAIL_FORGED_REPLYTO 2.25
score FREEMAIL_FROM 0.0
score FREEMAIL_REPLY 0.5
score FREEMAIL_REPLYTO 2.0
score FREEMAIL_REPLYTO_END_DIGIT 0.0
score FROM_LOCAL_DIGITS 1.75
score FROM_EXCESS_BASE64 2.5
score FROM_FMBLA_NEWDOM 2.5
score FROM_FMBLA_NEWDOM14 3.0
score FROM_FMBLA_NEWDOM28 2.5
score FROM_SUSPICIOUS_NTLD 2.0
score FORGED_GMAIL_RCVD 3.0
score FORGED_MUA_OUTLOOK 3.0
score FORGED_YAHOO_RCVD 3.0


# ============================
# Scam, Phishing & Social Engineering
# ============================

score ADVANCE_FEE_2 3.0
score ADVANCE_FEE_2_NEW_MONEY 3.0
score ADVANCE_FEE_2_NEW_FORM 3.0
score ADVANCE_FEE_3 3.0
score ADVANCE_FEE_3_NEW 3.0
score ADVANCE_FEE_3_NEW_MONEY 3.0
score ADVANCE_FEE_3_NEW_FORM 3.0
score ADVANCE_FEE_4_NEW 3.0
score ADVANCE_FEE_5_NEW 3.0
score ADVANCE_FEE_5_NEW_FRM_MNY 3.0
score ADVANCE_FEE_5_NEW_MONEY 3.0
score BILLION_DOLLARS 1.0
score DEAR_FRIEND 1.5
score LOTS_OF_MONEY 0.0
score UNCLAIMED_MONEY 4.0
score URG_BIZ 1.5
score FUZZY_CLICK_HERE 1.5
score FUZZY_CREDIT 2.0
score FUZZY_IMPORTANT 2.0
score FUZZY_SECURITY 2.25
score FUZZY_UNSUBSCRIBE 1.0
score SUBJ_AS_SEEN 0.75
score SUBJ_DOLLARS 0.25
score SUBJ_YOUR_DEBT 2.5
score SUBJ_YOUR_FAMILY 0.75


# ============================
# Transport / Network Reputation Rules
# ============================

score NO_RDNS_DOTCOM_HELO 3.0
score RDNS_DYNAMIC 3.5
score RDNS_LOCALHOST 3.5
score RDNS_NONE 2.25
score RCVD_HELO_IP_MISMATCH 1.75
score RCVD_ILLEGAL_IP 4.0
score PDS_BAD_THREAD_QP_64 1.5
score TBIRD_SUSP_MIME_BDRY 2.5
score SPAMMY_XMAILER 2.75
score KHOP_HELO_FCRDNS -0.5
score HELO_DYNAMIC_IPADDR 2.0
score HELO_DYNAMIC_SPLIT_IP 2.0
score UNPARSEABLE_RELAY 0.0


# ============================
# URI & Link Obfuscation
# ============================

score URI_HEX 2.5
score URI_NO_WWW_BIZ_CGI 2.5
score URI_NO_WWW_INFO_CGI 2.5
score URI_OBFU_WWW 3.0
score URI_TRUNCATED 2.5
score HTTPS_HTTP_MISMATCH 0.25
score WEIRD_PORT 4.5


# ============================
# Miscellaneous Heuristics & Content Triggers
# ============================

score ENV_AND_HDR_SPF_MATCH -4.0
score DATE_IN_FUTURE_06_12 2.25
score DATE_IN_PAST_03_06 2.25
score DATE_IN_PAST_06_12 2.25
score PLING_QUERY 1.0
score SHOPIFY_IMG_NOT_RCVD_SFY 0.75
score T_FILL_THIS_FORM_SHORT 0.25
score T_REMOTE_IMAGE 0.25
score STOX_REPLY_TYPE 2.0
score STOX_REPLY_TYPE_WITHOUT_QUOTES 3.0
score SUSPICIOUS_RECIPS 2.5


# ============================
# Spam Eating Monkey DNSBL lists
# ============================

header RCVD_IN_SEM_BACKSCATTER eval:check_rbl('sembackscatter-lastexternal', 'backscatter.spameatingmonkey.net')
describe RCVD_IN_SEM_BACKSCATTER Received from an IP listed by Spam Eating Monkey Backscatter list
tflags RCVD_IN_SEM_BACKSCATTER net
score RCVD_IN_SEM_BACKSCATTER 3.0

header RCVD_IN_SEM_BLACK eval:check_rbl('semblack-lastexternal', 'bl.spameatingmonkey.net')
tflags RCVD_IN_SEM_BLACK net
describe RCVD_IN_SEM_BLACK Received from an IP listed by Spam Eating Monkey Blocklist
score RCVD_IN_SEM_BLACK 3.0

header RCVD_IN_SEM_NETBLACK eval:check_rbl('semnetblack-lastexternal', 'netbl.spameatingmonkey.net')
tflags RCVD_IN_SEM_NETBLACK net
describe RCVD_IN_SEM_NETBLACK Received from an IP listed by Spam Eating Monkeys Network Blocklist
score RCVD_IN_SEM_NETBLACK 1.5

urirhssub SEM_FRESH30 fresh30.spameatingmonkey.net. A 2
body SEM_FRESH30 eval:check_uridnsbl('SEM_FRESH30')
describe SEM_FRESH30 Contains a domain registered less than 30 days ago
tflags SEM_FRESH30 net
score SEM_FRESH30 3.0

urirhssub SEM_URI_BLACK uribl.spameatingmonkey.net. A 2
body SEM_URI_BLACK eval:check_uridnsbl('SEM_URI')
describe SEM_URI_BLACK Contains a URI listed by Spam Eating Monkeys URI Blocklist
tflags SEM_URI_BLACK net
score SEM_URI_BLACK 3.5


# ============================
# JunkEmailFilter HostKarma DNSBL & DNSWL
# ============================

header __RCVD_IN_HOSTKARMA eval:check_rbl('hostkarma', 'hostkarma.junkemailfilter.com.')
describe __RCVD_IN_HOSTKARMA Sender listed in JunkEmailFilter
tflags __RCVD_IN_HOSTKARMA net

header RCVD_IN_HOSTKARMA_W eval:check_rbl_sub('hostkarma','127.0.0.1')
describe RCVD_IN_HOSTKARMA_W Sender listed in HOSTKARMA-WHITE
tflags RCVD_IN_HOSTKARMA_W net nice
score RCVD_IN_HOSTKARMA_W -3.5

header RCVD_IN_HOSTKARMA_BL eval:check_rbl_sub('hostkarma','127.0.0.2')
describe RCVD_IN_HOSTKARMA_BL Sender listed in HOSTKARMA-BLACK
tflags RCVD_IN_HOSTKARMA_BL net
score RCVD_IN_HOSTKARMA_BL 3.0

header RCVD_IN_HOSTKARMA_BR eval:check_rbl_sub('hostkarma','127.0.0.4')
describe RCVD_IN_HOSTKARMA_BR Sender listed in HOSTKARMA-BROWN
tflags RCVD_IN_HOSTKARMA_BR net
score RCVD_IN_HOSTKARMA_BR 1.25


# ============================
# SpamRATS DNSBL
# ============================

header __RCVD_IN_SPAMRATS eval:check_rbl('spamrats', 'all.spamrats.com.')
describe __RCVD_IN_SPAMRATS SPAMRATS: sender is listed in SpamRATS
tflags __RCVD_IN_SPAMRATS net
reuse __RCVD_IN_SPAMRATS

header RCVD_IN_SPAMRATS_DYNA eval:check_rbl_sub('spamrats','127.0.0.36')
describe RCVD_IN_SPAMRATS_DYNA RATS-Dyna: sent directly from dynamic IP address
tflags RCVD_IN_SPAMRATS_DYNA net
reuse RCVD_IN_SPAMRATS_DYNA
score RCVD_IN_SPAMRATS_DYNA 2.25

header RCVD_IN_SPAMRATS_NOPTR eval:check_rbl_sub('spamrats','127.0.0.37')
describe RCVD_IN_SPAMRATS_NOPTR RATS-NoPtr: sender has no reverse DNS
tflags RCVD_IN_SPAMRATS_NOPTR net
reuse RCVD_IN_SPAMRATS_NOPTR
score RCVD_IN_SPAMRATS_NOPTR 2.5

header RCVD_IN_SPAMRATS_SPAM eval:check_rbl_sub('spamrats','127.0.0.38')
describe RCVD_IN_SPAMRATS_SPAM RATS-Spam: sender is a spam source
tflags RCVD_IN_SPAMRATS_SPAM net
reuse RCVD_IN_SPAMRATS_SPAM
score RCVD_IN_SPAMRATS_SPAM 5.5


# ============================
# UCEPROTECT
# ============================

header RCVD_IN_UCEPROTECT_LEVEL_1 eval:check_rbl('uceprotect1', 'dnsbl-1.uceprotect.net.')
describe RCVD_IN_UCEPROTECT_LEVEL_1 Sender IP listed in UCEPROTECT Level 1
tflags RCVD_IN_UCEPROTECT_LEVEL_1 net
score RCVD_IN_UCEPROTECT_LEVEL_1 4.0

header RCVD_IN_UCEPROTECT_LEVEL_2 eval:check_rbl('uceprotect2', 'dnsbl-2.uceprotect.net.')
describe RCVD_IN_UCEPROTECT_LEVEL_2 Sender IP listed in UCEPROTECT Level 2
tflags RCVD_IN_UCEPROTECT_LEVEL_2 net
score RCVD_IN_UCEPROTECT_LEVEL_2 1.25


# ============================
# Abusix Guardian Mail Relay
# ============================

header __RCVD_IN_ABUSIX eval:check_rbl('abusix', '<API_KEY>.combined.mail.abusix.zone.')
describe __RCVD_IN_ABUSIX Received via a relay in Abusix Guardian Mail
tflags __RCVD_IN_ABUSIX net

header RCVD_IN_ABUSIX_BLACK eval:check_rbl_sub('abusix', '^127\.0\.0\.(?:[23]|200)$')
describe RCVD_IN_ABUSIX_BLACK Received via a relay in Abusix Guardian Mail Black
tflags RCVD_IN_ABUSIX_BLACK net
score RCVD_IN_ABUSIX_BLACK 5.5

header RCVD_IN_ABUSIX_EXPLOIT eval:check_rbl_sub('abusix', '127.0.0.4')
describe RCVD_IN_ABUSIX_EXPLOIT Received via a relay in Abusix Guardian Mail Exploit
tflags RCVD_IN_ABUSIX_EXPLOIT net
score RCVD_IN_ABUSIX_EXPLOIT 6.0

header RCVD_IN_ABUSIX_DYN eval:check_rbl('abusix_dyn', '<API_KEY>.combined.mail.abusix.zone.', '^127\.0\.0\.1[12]$')
describe RCVD_IN_ABUSIX_DYN Received via a relay in Abusix Guardian Mail Dynamic
tflags RCVD_IN_ABUSIX_DYN net
score RCVD_IN_ABUSIX_DYN 1.5

header RCVD_IN_ABUSIX_WHITE eval:check_rbl('abusix_white', '<API_KEY>.combined.mail.abusix.zone.', '127.0.2.1')
describe RCVD_IN_ABUSIX_WHITE Received via a relay in Abusix Guardian Mail White
tflags RCVD_IN_ABUSIX_WHITE nice net
score RCVD_IN_ABUSIX_WHITE -1.5

urirhsbl URIBL_ABUSIX_DBLACK <API_KEY>.dblack.mail.abusix.zone. A
body URIBL_ABUSIX_DBLACK eval:check_uridnsbl('URIBL_ABUSIX_DBLACK')
describe URIBL_ABUSIX_DBLACK Contains a spam URL listed in the Abusix domain blocklist
tflags URIBL_ABUSIX_DBLACK net
score URIBL_ABUSIX_DBLACK 6.0

urirhssub URIBL_ABUSIX_WHITE <API_KEY>.white.mail.abusix.zone. A 127.0.2.1
body URIBL_ABUSIX_WHITE eval:check_uridnsbl('URIBL_ABUSIX_WHITE')
describe URIBL_ABUSIX_WHITE Contains a domain listed in the Abusix domain whitelist
tflags URIBL_ABUSIX_WHITE nice net
score URIBL_ABUSIX_WHITE -0.25


# ============================
# Ascams RBLs (IP Reputation)
# ============================

header RCVD_IN_ASCAMS_BLOCK eval:check_rbl('ascams_block', 'block.ascams.com.')
describe RCVD_IN_ASCAMS_BLOCK Sender listed in Ascams Block RBL
tflags RCVD_IN_ASCAMS_BLOCK net
score RCVD_IN_ASCAMS_BLOCK 1.25

header RCVD_IN_ASCAMS_DROP eval:check_rbl('ascams_white', 'dnsbl.ascams.com.')
describe RCVD_IN_ASCAMS_DROP Sender listed in Ascams DROP list
tflags RCVD_IN_ASCAMS_DROP nice net
score RCVD_IN_ASCAMS_DROP 3.0


# ============================
# DroneBL DNSBL
# ============================

header RCVD_IN_DRONEBL eval:check_rbl('dronebl', 'dnsbl.dronebl.org.')
describe RCVD_IN_DRONEBL Sender listed in DroneBL (suspected bot/malware)
tflags RCVD_IN_DRONEBL net
score RCVD_IN_DRONEBL 3.0


# ============================
# GBUDB Truncate DNSBL
# ============================

header RCVD_IN_GBUDB_TRUNCATE eval:check_rbl('gbudb', 'truncate.gbudb.net.')
describe RCVD_IN_GBUDB_TRUNCATE Sender listed in GBUDB Truncate
tflags RCVD_IN_GBUDB_TRUNCATE net
score RCVD_IN_GBUDB_TRUNCATE 4.0


# ============================
# Usenix S5H
# ============================

header RCVD_IN_S5H_BL eval:check_rbl_txt('s5hbl', 'all.s5h.net.')
describe RCVD_IN_S5H_BL Listed at all.s5h.net
tflags RCVD_IN_S5H_BL net
score RCVD_IN_S5H_BL 0.5


# ============================
# Backscatterer.org 
# ============================

header RCVD_IN_BACKSCATTERER eval:check_rbl('backscatterer', 'ips.backscatterer.org.')
describe RCVD_IN_BACKSCATTERER IP listed in Backscatterer (backscatter spam)
tflags RCVD_IN_BACKSCATTERER net
score RCVD_IN_BACKSCATTERER 3.5

Not sure if this is related or not, but I noticed when I SSH'd into my server, and I noticed there was mention of a zombie process. I ran this and found it was node... I know Cloudron runs on node... is this possibly related at all?

ubuntu@my:~$ ps aux | grep Z
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
ubuntu   1999826  0.0  0.0      0     0 ?        Zs   03:25   0:00 [node] <defunct>

I wonder if this is perhaps related to the issue being troubleshot here: https://forum.cloudron.io/post/104263

@nebulon - tl;dr = The commands didn't change the behaviour at all, it was still causing backups to fail and Services page showed it as orange still.

--

When I restart the service affected (redis in this case) by exec'ing into the container and running supervisorctl restart redis-service, it doesn't solve the problem. It still suggests that redis is restarting correctly as the output shows the following:

root@redis-<UUID>:/app/code# supervisorctl restart redis-service
redis-service: stopped
redis-service: started

And the container logs show the following:

Apr 03 19:35:53 2025-04-04 02:35:53,254 INFO waiting for redis-service to stop
Apr 03 19:35:53 2025-04-04 02:35:53,256 WARN stopped: redis-service (terminated by SIGTERM)
Apr 03 19:35:53 2025-04-04 02:35:53,258 INFO spawned: 'redis-service' with pid 321
Apr 03 19:35:53 Redis service endpoint listening on http://:::3000
Apr 03 19:35:54 2025-04-04 02:35:54,320 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

However the services list still shows yellow and the backups continue to fail.

If I try with the following:

root@redis-<UUID>:/app/code# supervisorctl restart redis    
redis: stopped
redis: started

It's the same behaviour, however I do see a little more in the logs for the container:

Apr 03 19:40:08 2025-04-04 02:40:08,186 INFO waiting for redis to stop
Apr 03 19:40:08 330:signal-handler (1743734408) Received SIGTERM scheduling shutdown...
Apr 03 19:40:08 330:M 04 Apr 2025 02:40:08.208 * User requested shutdown...
Apr 03 19:40:08 330:M 04 Apr 2025 02:40:08.208 * Saving the final RDB snapshot before exiting.
Apr 03 19:40:08 330:M 04 Apr 2025 02:40:08.216 * DB saved on disk
Apr 03 19:40:08 330:M 04 Apr 2025 02:40:08.216 * Removing the pid file.
Apr 03 19:40:08 330:M 04 Apr 2025 02:40:08.216 # Redis is now ready to exit, bye bye...
Apr 03 19:40:08 2025-04-04 02:40:08,218 INFO stopped: redis (exit status 0)
Apr 03 19:40:08 2025-04-04 02:40:08,220 INFO spawned: 'redis' with pid 337
Apr 03 19:40:08 337:C 04 Apr 2025 02:40:08.225 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Apr 03 19:40:08 337:C 04 Apr 2025 02:40:08.225 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Apr 03 19:40:08 337:C 04 Apr 2025 02:40:08.225 * Redis version=7.4.2, bits=64, commit=00000000, modified=0, pid=337, just started
Apr 03 19:40:08 337:C 04 Apr 2025 02:40:08.225 * Configuration loaded
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.225 * monotonic clock: POSIX clock_gettime
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 # Failed to write PID file: Permission denied
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 * Running mode=standalone, port=6379.
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 * Server initialized
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 * Loading RDB produced by version 7.4.2
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 * RDB age 0 seconds
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.226 * RDB memory usage when created 2.70 Mb
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.230 * Done loading RDB, keys loaded: 2177, keys expired: 0.
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.230 * DB loaded from disk: 0.004 seconds
Apr 03 19:40:08 337:M 04 Apr 2025 02:40:08.230 * Ready to accept connections tcp
Apr 03 19:40:09 2025-04-04 02:40:09,231 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

What I was able to do though temporarily to get this to work is restart the docker service entirely, but of course this method tends to cause a little bit of downtime. This should be stable until the next time I need to restart my server for security updates, I suspect, but this really needs to get figured out. If nobody else is seeing this, perhaps it's something unique to my server? I did upgrade to Ubuntu 24.04 a few weeks ago but at the same time I also upgraded to the latest Cloudron version too, so it's hard to know if either of those or which one (if at all) is possibly related to the issue, but something to keep in mind in case you think these are related at all.

What else can we try?

@nebulon I'm not sure how to tell which redis container is the culprit. The Cloudron Services page shows which one, but there's no container ID associated with it from the UI that I can find. And in the docker ps command, everything shows "up 22 hours" for example, nothing looks wrong from that list. Is there a way to know which one to exec into?

Edit: Sorry, I figured it out. Looking at the logs of the affected container shows the container name.

Hi @girish , I'm still having this issue intermittently, but the logs really don't seem to suggest anything. When these services become affected, it causes the backups to fail which is very concerning.

Right now, the logs don't seem to show anything. Here's the latest affected redis instance (it rotates around by the way, it is random which redis container seems to get impacted):

Apr 02 17:37:36 ==> Starting supervisor
Apr 02 17:37:36 2025-04-03 00:37:36,731 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
Apr 02 17:37:36 2025-04-03 00:37:36,731 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
Apr 02 17:37:36 2025-04-03 00:37:36,731 INFO Set uid to user 0 succeeded
Apr 02 17:37:36 2025-04-03 00:37:36,734 INFO RPC interface 'supervisor' initialized
Apr 02 17:37:36 2025-04-03 00:37:36,734 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Apr 02 17:37:36 2025-04-03 00:37:36,734 INFO supervisord started with pid 1
Apr 02 17:37:37 2025-04-03 00:37:37,736 INFO spawned: 'redis' with pid 13
Apr 02 17:37:37 2025-04-03 00:37:37,736 INFO spawned: 'redis-service' with pid 14
Apr 02 17:37:37 13:C 03 Apr 2025 00:37:37.741 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Apr 02 17:37:37 13:C 03 Apr 2025 00:37:37.741 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Apr 02 17:37:37 13:C 03 Apr 2025 00:37:37.741 * Redis version=7.4.2, bits=64, commit=00000000, modified=0, pid=13, just started
Apr 02 17:37:37 13:C 03 Apr 2025 00:37:37.741 * Configuration loaded
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.741 * monotonic clock: POSIX clock_gettime
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 # Failed to write PID file: Permission denied
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 * Running mode=standalone, port=6379.
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 * Server initialized
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 * Loading RDB produced by version 7.4.2
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 * RDB age 1 seconds
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.742 * RDB memory usage when created 2.08 Mb
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.744 * Done loading RDB, keys loaded: 1513, keys expired: 0.
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.744 * DB loaded from disk: 0.003 seconds
Apr 02 17:37:37 13:M 03 Apr 2025 00:37:37.744 * Ready to accept connections tcp
Apr 02 17:37:37 Redis service endpoint listening on http://:::3000
Apr 02 17:37:38 2025-04-03 00:37:38,802 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 02 17:37:38 2025-04-03 00:37:38,802 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 02 17:42:14 2025-04-03 00:42:14,099 WARN received SIGTERM indicating exit request
Apr 02 17:42:14 2025-04-03 00:42:14,099 INFO waiting for redis, redis-service to die
Apr 02 17:42:14 2025-04-03 00:42:14,101 WARN stopped: redis-service (terminated by SIGTERM)
Apr 02 17:42:14 13:signal-handler (1743640934) Received SIGTERM scheduling shutdown...
Apr 02 17:42:14 13:M 03 Apr 2025 00:42:14.177 * User requested shutdown...
Apr 02 17:42:14 13:M 03 Apr 2025 00:42:14.177 * Saving the final RDB snapshot before exiting.
Apr 02 17:42:14 13:M 03 Apr 2025 00:42:14.185 * DB saved on disk
Apr 02 17:42:14 13:M 03 Apr 2025 00:42:14.185 * Removing the pid file.
Apr 02 17:42:14 13:M 03 Apr 2025 00:42:14.185 # Redis is now ready to exit, bye bye...
Apr 02 17:42:14 2025-04-03 00:42:14,186 INFO stopped: redis (exit status 0)
Apr 02 17:42:28 ==> Starting supervisor
Apr 02 17:42:28 2025-04-03 00:42:28,750 INFO Included extra file "/etc/supervisor/conf.d/redis-service.conf" during parsing
Apr 02 17:42:28 2025-04-03 00:42:28,750 INFO Included extra file "/etc/supervisor/conf.d/redis.conf" during parsing
Apr 02 17:42:28 2025-04-03 00:42:28,750 INFO Set uid to user 0 succeeded
Apr 02 17:42:28 2025-04-03 00:42:28,756 INFO RPC interface 'supervisor' initialized
Apr 02 17:42:28 2025-04-03 00:42:28,756 CRIT Server 'inet_http_server' running without any HTTP authentication checking
Apr 02 17:42:28 2025-04-03 00:42:28,756 INFO supervisord started with pid 1
Apr 02 17:42:29 2025-04-03 00:42:29,759 INFO spawned: 'redis' with pid 13
Apr 02 17:42:29 2025-04-03 00:42:29,761 INFO spawned: 'redis-service' with pid 14
Apr 02 17:42:29 13:C 03 Apr 2025 00:42:29.779 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
Apr 02 17:42:29 13:C 03 Apr 2025 00:42:29.779 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
Apr 02 17:42:29 13:C 03 Apr 2025 00:42:29.779 * Redis version=7.4.2, bits=64, commit=00000000, modified=0, pid=13, just started
Apr 02 17:42:29 13:C 03 Apr 2025 00:42:29.779 * Configuration loaded
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.779 * monotonic clock: POSIX clock_gettime
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 # Failed to write PID file: Permission denied
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 * Running mode=standalone, port=6379.
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 * Server initialized
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 * Loading RDB produced by version 7.4.2
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 * RDB age 15 seconds
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.780 * RDB memory usage when created 2.10 Mb
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.785 * Done loading RDB, keys loaded: 1513, keys expired: 0.
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.785 * DB loaded from disk: 0.005 seconds
Apr 02 17:42:29 13:M 03 Apr 2025 00:42:29.785 * Ready to accept connections tcp
Apr 02 17:42:29 Redis service endpoint listening on http://:::3000
Apr 02 17:42:30 2025-04-03 00:42:30,863 INFO success: redis entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 02 17:42:30 2025-04-03 00:42:30,863 INFO success: redis-service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
Apr 02 17:42:32 [GET] /healthcheck
Apr 02 17:42:33 [GET] /healthcheck
Apr 02 17:42:38 [GET] /healthcheck
Apr 02 17:42:45 [GET] /healthcheck
Apr 02 17:42:56 [GET] /healthcheck
Apr 02 17:47:28 [GET] /healthcheck
Apr 02 17:47:30 13:M 03 Apr 2025 00:47:30.072 * 10 changes in 300 seconds. Saving...
Apr 02 17:47:30 13:M 03 Apr 2025 00:47:30.072 * Background saving started by pid 56
Apr 02 17:47:30 56:C 03 Apr 2025 00:47:30.079 * DB saved on disk
Apr 02 17:47:30 56:C 03 Apr 2025 00:47:30.080 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 0 MB
Apr 02 17:47:30 13:M 03 Apr 2025 00:47:30.172 * Background saving terminated with success
Apr 02 17:47:34 [GET] /healthcheck
Apr 02 17:47:40 [GET] /healthcheck
Apr 02 17:47:45 [GET] /healthcheck
Apr 02 17:48:21 [POST] /backup
Apr 02 17:48:21 backing up
Apr 02 17:48:21 13:M 03 Apr 2025 00:48:21.512 * DB saved on disk
Apr 02 17:48:45 [GET] /healthcheck

The only thing I can do to temporarily get it into a good state is restart the Docker service, and I often need to do that a few times before it really becomes stable.

The dmesg output I captured for you and put it here (it was too long to put into this post): https://paste.cloudron.io/raw/utedesasid

I hope that helps. Some of the data is redacted in the public view of it just to be cautious, but I have the original file un-redacted if you need it and I can email it to you too.

For the backup failure part, in case that helps show context...

Apr 02 14:05:34 box:services backupAddons
Apr 02 14:05:34 box:services backupAddons: backing up ["mysql","oidc","localstorage","sendmail","scheduler","redis"]
Apr 02 14:05:34 box:services Backing up mysql
Apr 02 14:05:34 box:services pipeRequestToFile: connected with status code 200
Apr 02 14:05:34 box:services Backing up redis
Apr 02 14:05:39 box:services pipeRequestToFile: timeout - connect or post-connect idle timeout
Apr 02 14:05:39 box:backuptask fullBackup: app <domain> backup finished. Took 5.129 seconds
Apr 02 14:05:39 box:locks write: current locks: {"backup_task":null}
Apr 02 14:05:39 box:locks release: app_6120f84a-0953-4982-9167-ef872c0ee49a
Apr 02 14:05:39 box:taskworker Task took 339.377 seconds
Apr 02 14:05:39 box:tasks setCompleted - 28833: {"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1347:20)\n at Object.onceWrapper (node:events:633:28)\n at ClientRequest.emit (node:events:519:28)\n at ClientRequest.emit (node:domain:488:12)\n at Socket.emitRequestTimeout (node:_http_client:849:9)\n at Object.onceWrapper (node:events:633:28)\n at Socket.emit (node:events:531:35)\n at Socket.emit (node:domain:488:12)\n at Socket._onTimeout (node:net:591:8)\n at listOnTimeout (node:internal/timers:581:17)","message":"Request timedout"}}
Apr 02 14:05:39 box:tasks update 28833: {"percent":100,"result":null,"error":{"stack":"Error: Request timedout\n at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1347:20)\n at Object.onceWrapper (node:events:633:28)\n at ClientRequest.emit (node:events:519:28)\n at ClientRequest.emit (node:domain:488:12)\n at Socket.emitRequestTimeout (node:_http_client:849:9)\n at Object.onceWrapper (node:events:633:28)\n at Socket.emit (node:events:531:35)\n at Socket.emit (node:domain:488:12)\n at Socket._onTimeout (node:net:591:8)\n at listOnTimeout (node:internal/timers:581:17)","message":"Request timedout"}}
Apr 02 14:05:39 Error: Request timedout
Apr 02 14:05:39 at ClientRequest.<anonymous> (/home/yellowtent/box/src/services.js:1347:20)
Apr 02 14:05:39 at Object.onceWrapper (node:events:633:28)
Apr 02 14:05:39 at ClientRequest.emit (node:events:519:28)
Apr 02 14:05:39 at ClientRequest.emit (node:domain:488:12)
Apr 02 14:05:39 at Socket.emitRequestTimeout (node:_http_client:849:9)
Apr 02 14:05:39 at Object.onceWrapper (node:events:633:28)
Apr 02 14:05:39 at Socket.emit (node:events:531:35)
Apr 02 14:05:39 at Socket.emit (node:domain:488:12)
Apr 02 14:05:39 at Socket._onTimeout (node:net:591:8)
Apr 02 14:05:39 at listOnTimeout (node:internal/timers:581:17)

It seems like it's working again, but it's unclear why this is.