Firewall / Spamassassin: Automatic list update
-
@necrevistonnezr you can still use the api though
-
Well, the "setBlockList" operation allows to add a range of IPs but not a list of IPs in a file or am I wrong?
curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $CLOUDRON_TOKEN" "https://$CLOUDRON_DOMAIN/api/v1/network/blocklist" --data '{"blocklist":"# Spammy network\n10.244.0.0/16"}'
as per: https://docs.cloudron.io/api.html#tag/Network/operation/setBlockList
-
@necrevistonnezr it's a "Newline separated list of IP entries" . So, it can be
# Spammy network\n10.244.0.0/16\n1.2.3.4\n3.4.5.6\n172.4.0.0/16
-
@girish I guess there's no mechanism to avoid duplicate entries when using the "setBlockList" operation, correct?
In general, I guess something like this should work:
#!/bin/bash curl https://www.ipdeny.com/ipblocks/data/countries/kz.zone --output iplist.txt while read -r line; do curl -X POST -H "Content-Type: application/json" -H "Authorization: Bearer $CLOUDRON_TOKEN" "https://$CLOUDRON_DOMAIN/api/v1/network/blocklist" --data $line" done < iplist.txt rm iplist.txt
I don't have know yet how to avoid duplicates in the database..
-
It should be default functionality to have country block/allow in the Cloudron GUI just like all Synology NAS’s have. It’s 2023 and too dangerous to have everything accessible for everyone. That’s why many Cloudron users (read the forum) are using Cloudflare for this kind of functionality (like I have to do).
-
-
@imc67 IMO, the correct place to implement this is in the network firewall. Most Cloud providers already have a firewall feature and they can then implement this firewall rule at the edge of the network instead of the server itself.
I have a Synology router (not NAS) at home. I just use their blocklists. For home setups, the router is the correct place for this. Otherwise, you allow all traffic to come into your home and then it gets rejected by the server wasting cpu and network traffic.
That said, I understand why this feature is being requested here instead - no cloud network firewall has this feature. And most likely cloud providers don't listen to our suggestions
-
@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:
I guess something like this should work
Did you managed it to get it worked like that?
-
@necrevistonnezr I gave it a try with some help by ChatGPT and it works flawless!! Except the API can't handle large list where the GUI is able to handle without an issue.
The script automatically downloads all the geo lists in an array, creates a copy/paste file for the GUI and then prepares the file in JSON style and connect/upload via API.
When I choose only a few countries is works perfect, however when choosing all the desired ones:
@girish @nebulon I get a
line 83: /usr/bin/curl: Argument list too long
I can copy/paste the full list in the GUI, it takes some time but it uploads and settles all IP ranges (about 87k)
Does anyone know how to do this via the API?
-
@imc67 said in Firewall / Spamassassin: Automatic list update:
@girish @nebulon I get a line 83: /usr/bin/curl: Argument list too long
this is related to
curl
. I don't know the answer but you can look for posts similar to https://stackoverflow.com/questions/54090784/curl-argument-list-too-long where you have to pass the args as a file instead of on the command line itself. -
@girish said in Firewall / Spamassassin: Automatic list update:
https://stackoverflow.com/questions/54090784/curl-argument-list-too-long
I tried that in the beginning but got this error:
{ "status": "Bad Request", "message": "blocklist must be a string" }
-
Again ChatGPT did it!!! It's really unbelievable how "patient" it is and after keep trying all it's variants (after feedback the errors) and asking if there is another way except curl it came with wget and after the second try it worked!
-
use at your own risk and be aware that with this long list it takes 1,5 minute to process the API call (same as in the GUI), replace the API input and copy/paste it in a .sh file (don't forget to make it executable). It's works but it's not perfect.
#!/bin/bash # Huidige datum en tijd current_datetime=$(date +"%Y%m%d_%H%M%S") # Array met de URL's van de IP-lijsten en hun beschrijvingen declare -a urls=( "https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop" "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop" "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipblocks/data/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipblocks/data/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipblocks/data/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipblocks/data/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipblocks/data/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipblocks/data/aggregated/kp-aggregated.zone,KP - North Korea" "https://www.ipdeny.com/ipblocks/data/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipblocks/data/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipblocks/data/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipblocks/data/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipblocks/data/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipblocks/data/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipblocks/data/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipblocks/data/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipblocks/data/aggregated/vn-aggregated.zone,VN - Vietnam" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/vn-aggregated.zone,VN - Vietnam" ) # Bestandsnaam met de huidige datum en tijd output_file="samengevoegde_lijst_${current_datetime}.txt" # Downloaden en samenvoegen van de IP-lijsten for url_info in "${urls[@]}" do # Opsplitsen van de URL-informatie IFS=',' read -r url description <<< "$url_info" # Opmerking toevoegen met de URL en beschrijving echo "# URL: $url" >> "$output_file" echo "# Description: $description" >> "$output_file" echo "IP-lijst downloaden van $url" # Downloaden van de IP-lijst en toevoegen aan het bestand curl -sS "$url" >> "$output_file" done echo "Samenvoegen voltooid! De samengevoegde lijst is opgeslagen in $output_file" # Formatteren van het bestand voor de Cloudron Blocklist API formatted_file="formatted_$output_file" # Voeg "\n" toe aan het einde van elke regel awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file" # Cloudron Blocklist API endpoint cloudron_api_endpoint="https://your-cloudron-domain.com/api/v1/network/blocklist" # API Key voor authenticatie (vervang 'your-api-key' door jouw API-sleutel) api_key="your-api-key" # Uploaden naar Cloudron Blocklist API met wget echo "Uploaden naar Cloudron Blocklist API met wget..." # De gegevens in het vereiste formaat voor de API data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}" # Verzend het bestand met een POST-verzoek via wget echo "$data" > temp_data.txt wget --method=POST --header="Content-Type: application/json" --header="Authorization: Bearer $api_key" --body-file=temp_data.txt "$cloudron_api_endpoint" --quiet --output-document=output.txt # Toon de uitvoer van wget cat output.txt # Verwijder tijdelijke bestanden rm temp_data.txt
-
-
Every time you execute the script the existing list is replaced by the newly generated version. There might be duplicates at generating the list because IP’s from the first two URL’s can exist in the later ones but that’s not a problem for me.
-
Thanks again!
I just tried this - it generatedmerged_list_20240110_091244.txt
andformatted_merged_list_20240110_091244.txt
but somehow, thetemp_data.txt
is not generated, and therefore the upload is not successful?BTW, here's your script with english comments and filenames:
#!/bin/bash # Current date and time current_datetime=$(date +"%Y%m%d_%H%M%S") # Array containing the URLs of the IP lists and their descriptions declare -a urls=( "https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop" "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop" "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipblocks/data/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipblocks/data/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipblocks/data/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipblocks/data/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipblocks/data/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipblocks/data/aggregated/kp-aggregated.zone,KP - North Korea" "https://www.ipdeny.com/ipblocks/data/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipblocks/data/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipblocks/data/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipblocks/data/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipblocks/data/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipblocks/data/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipblocks/data/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipblocks/data/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipblocks/data/aggregated/vn-aggregated.zone,VN - Vietnam" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/vn-aggregated.zone,VN - Vietnam" ) # File name with the current date and time output_file="merged_list_${current_datetime}.txt" # Download and merge the IP lists for url_info in "${urls[@]}" do # Splitting the URL information IFS=',' read -r url description <<< "$url_info" # Add comment with the URL and description echo "# URL: $url" >> "$output_file" echo "# Description: $description" >> "$output_file" echo "Download IP list from $url" # Download the IP list and add it to the file curl -sS "$url" >> "$output_file" done echo "Merge completed! The merged list is stored in $output_file" # Formatting the file for the Cloudron Blocklist API formatted_file="formatted_$output_file" # # Add "\n" to the end of each line awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file" # Cloudron Blocklist API endpoint cloudron_api_endpoint="https://yourcloudrondomain.com/api/v1/network/blocklist" # API Key for authentication (replace 'your-api-key' with your API key) api_key="your-api-key" # Upload to Cloudron Blocklist API with wget echo "# Upload to Cloudron Blocklist API with wget..." # The data in the required format for the API data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}" # Send the file with a POST request via wget echo "$data"> temp_data.txt wget --method=POST --header="Content-Type: application/json" --header="Authorization: Bearer $api_key" --body-file=temp_data.txt "$cloudron_api_endpoint" --quiet --output-document=output.txt # Show the output of wget cat output.txt # Delete temporary files rm temp_data.txt