Crashed Cloudron server when adding in many IP address ranges (via Firehol) to the Cloudron Network Blocklist
-
I was adding in all the IP address ranges as a test from the file located at https://iplists.firehol.org/files/firehol_level1.netset which contains over 2000 line entries. I probably should have known this would cause some overload, but I wasn't expecting it to completely crash causing me to need to reboot my entire server. I'm currently running into issues running Cloudron and trying to recover right now. Here are the latest logs I could find:
2021-11-15T05:40:11.246Z box:shell setBlocklist spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/setblocklist.sh 2021-11-15T05:42:14.009Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:42:15.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:42:16.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:42:17.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:42:17.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:43:14.008Z box:cron BoxError: connect ETIMEDOUT at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38) at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13 at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9) at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10) at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16) at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14) at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18) at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8) 2021-11-15T05:43:15.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:43:16.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:43:17.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:43:17.008Z box:cron BoxError: connect ETIMEDOUT at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38) at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13 at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9) at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10) at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16) at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14) at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18) at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8) 2021-11-15T05:44:14.009Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:44:15.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:44:16.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:44:17.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:44:17.008Z box:scheduler could not run task wpcron : connect ETIMEDOUT 2021-11-15T05:45:14.011Z box:cron BoxError: connect ETIMEDOUT at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38) at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13 at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9) at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10) at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16) at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24) at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14) at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18) at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8)
-
Resolved by following the steps below, hopefully this resolves it for anyone else who may run into this issue:
- Delete the
/home/yellowtent/platformdata/firewall/blocklist.txt
file - Reboot the server
This seemed to resolve the issue, though what was a little strange (but maybe this was just cache?) was the Network tab later showed it was blocking well over 2000 IP addresses which in my mind seemed like it shouldn't have been the case since we deleted the blocklist, but I then deleted it from the Network tab just in case. Looking back, unsure if that was more of a caching thing or not.
Simply rebooting the server itself didn't resolve the issue in my case, it seemed I needed to reset that list for the firewall to make things work again.
The idea of deleting the blocklist.txt file came from here: https://docs.cloudron.io/networking/#blocklist -- while I wasn't locked out (my IP wasn't on the list or in the range), it did seem adequate since I was modifying the blocklist by adding in a couple thousand entries at once that it'd be a matter of just needing to reset that value back to what it was prior by removing that file located in the docs.
- Delete the