Crashed Cloudron server when adding in many IP address ranges (via Firehol) to the Cloudron Network Blocklist

[d19dotca]

I was adding in all the IP address ranges as a test from the file located at https://iplists.firehol.org/files/firehol_level1.netset which contains over 2000 line entries. I probably should have known this would cause some overload, but I wasn't expecting it to completely crash causing me to need to reboot my entire server. I'm currently running into issues running Cloudron and trying to recover right now. Here are the latest logs I could find:

2021-11-15T05:40:11.246Z box:shell setBlocklist spawn: /usr/bin/sudo -S /home/yellowtent/box/src/scripts/setblocklist.sh
2021-11-15T05:42:14.009Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:42:15.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:42:16.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:42:17.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:42:17.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:43:14.008Z box:cron BoxError: connect ETIMEDOUT
    at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38)
    at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13
    at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9)
    at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)
    at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)
    at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14)
    at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18)
    at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8)
2021-11-15T05:43:15.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:43:16.005Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:43:17.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:43:17.008Z box:cron BoxError: connect ETIMEDOUT
    at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38)
    at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13
    at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9)
    at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)
    at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)
    at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14)
    at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18)
    at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8)
2021-11-15T05:44:14.009Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:44:15.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:44:16.006Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:44:17.007Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:44:17.008Z box:scheduler could not run task wpcron : connect ETIMEDOUT
2021-11-15T05:45:14.011Z box:cron BoxError: connect ETIMEDOUT
    at Query.queryCallback [as _callback] (/home/yellowtent/box/src/database.js:91:38)
    at Query.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at /home/yellowtent/box/node_modules/mysql/lib/Pool.js:205:13
    at Handshake.onConnect (/home/yellowtent/box/node_modules/mysql/lib/Pool.js:58:9)
    at Handshake.<anonymous> (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:526:10)
    at Handshake._callback (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:488:16)
    at Handshake.Sequence.end (/home/yellowtent/box/node_modules/mysql/lib/protocol/sequences/Sequence.js:83:24)
    at Protocol.handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/protocol/Protocol.js:369:14)
    at PoolConnection.Connection._handleNetworkError (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:418:18)
    at PoolConnection.Connection._handleConnectTimeout (/home/yellowtent/box/node_modules/mysql/lib/Connection.js:414:8)

[d19dotca]

Resolved by following the steps below, hopefully this resolves it for anyone else who may run into this issue:

1. Delete the /home/yellowtent/platformdata/firewall/blocklist.txt file
2. Reboot the server

This seemed to resolve the issue, though what was a little strange (but maybe this was just cache?) was the Network tab later showed it was blocking well over 2000 IP addresses which in my mind seemed like it shouldn't have been the case since we deleted the blocklist, but I then deleted it from the Network tab just in case. Looking back, unsure if that was more of a caching thing or not.

Simply rebooting the server itself didn't resolve the issue in my case, it seemed I needed to reset that list for the firewall to make things work again.

The idea of deleting the blocklist.txt file came from here: https://docs.cloudron.io/networking/#blocklist -- while I wasn't locked out (my IP wasn't on the list or in the range), it did seem adequate since I was modifying the blocklist by adding in a couple thousand entries at once that it'd be a matter of just needing to reset that value back to what it was prior by removing that file located in the docs.

[JOduMonT]

@d19dotca yes firehol is very interesting
but also very dangerous, especially with IPTables
our mind crossed each other, I just mentionned FireHol in a request: https://forum.cloudron.io/topic/6009/firewall-per-domain-container