Firewall / Spamassassin: Automatic list update
-
use at your own risk and be aware that with this long list it takes 1,5 minute to process the API call (same as in the GUI), replace the API input and copy/paste it in a .sh file (don't forget to make it executable). It's works but it's not perfect.
#!/bin/bash # Huidige datum en tijd current_datetime=$(date +"%Y%m%d_%H%M%S") # Array met de URL's van de IP-lijsten en hun beschrijvingen declare -a urls=( "https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop" "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop" "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipblocks/data/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipblocks/data/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipblocks/data/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipblocks/data/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipblocks/data/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipblocks/data/aggregated/kp-aggregated.zone,KP - North Korea" "https://www.ipdeny.com/ipblocks/data/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipblocks/data/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipblocks/data/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipblocks/data/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipblocks/data/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipblocks/data/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipblocks/data/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipblocks/data/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipblocks/data/aggregated/vn-aggregated.zone,VN - Vietnam" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/vn-aggregated.zone,VN - Vietnam" ) # Bestandsnaam met de huidige datum en tijd output_file="samengevoegde_lijst_${current_datetime}.txt" # Downloaden en samenvoegen van de IP-lijsten for url_info in "${urls[@]}" do # Opsplitsen van de URL-informatie IFS=',' read -r url description <<< "$url_info" # Opmerking toevoegen met de URL en beschrijving echo "# URL: $url" >> "$output_file" echo "# Description: $description" >> "$output_file" echo "IP-lijst downloaden van $url" # Downloaden van de IP-lijst en toevoegen aan het bestand curl -sS "$url" >> "$output_file" done echo "Samenvoegen voltooid! De samengevoegde lijst is opgeslagen in $output_file" # Formatteren van het bestand voor de Cloudron Blocklist API formatted_file="formatted_$output_file" # Voeg "\n" toe aan het einde van elke regel awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file" # Cloudron Blocklist API endpoint cloudron_api_endpoint="https://your-cloudron-domain.com/api/v1/network/blocklist" # API Key voor authenticatie (vervang 'your-api-key' door jouw API-sleutel) api_key="your-api-key" # Uploaden naar Cloudron Blocklist API met wget echo "Uploaden naar Cloudron Blocklist API met wget..." # De gegevens in het vereiste formaat voor de API data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}" # Verzend het bestand met een POST-verzoek via wget echo "$data" > temp_data.txt wget --method=POST --header="Content-Type: application/json" --header="Authorization: Bearer $api_key" --body-file=temp_data.txt "$cloudron_api_endpoint" --quiet --output-document=output.txt # Toon de uitvoer van wget cat output.txt # Verwijder tijdelijke bestanden rm temp_data.txt
-
N necrevistonnezr referenced this topic on
-
Every time you execute the script the existing list is replaced by the newly generated version. There might be duplicates at generating the list because IP’s from the first two URL’s can exist in the later ones but that’s not a problem for me.
-
Thanks again!
I just tried this - it generatedmerged_list_20240110_091244.txt
andformatted_merged_list_20240110_091244.txt
but somehow, thetemp_data.txt
is not generated, and therefore the upload is not successful?BTW, here's your script with english comments and filenames:
#!/bin/bash # Current date and time current_datetime=$(date +"%Y%m%d_%H%M%S") # Array containing the URLs of the IP lists and their descriptions declare -a urls=( "https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop" "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop" "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipblocks/data/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipblocks/data/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipblocks/data/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipblocks/data/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipblocks/data/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipblocks/data/aggregated/kp-aggregated.zone,KP - North Korea" "https://www.ipdeny.com/ipblocks/data/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipblocks/data/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipblocks/data/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipblocks/data/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipblocks/data/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipblocks/data/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipblocks/data/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipblocks/data/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipblocks/data/aggregated/vn-aggregated.zone,VN - Vietnam" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/bd-aggregated.zone,BD - Bangladesh" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/by-aggregated.zone,BY - Belarus" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/br-aggregated.zone,BR - Brazil" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/cn-aggregated.zone,CN - China" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ir-aggregated.zone,IR - Iran" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/in-aggregated.zone,IN - India" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/np-aggregated.zone,NP - Nepal" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/pk-aggregated.zone,PK - Pakistan" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ro-aggregated.zone,RO - Romania" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ru-aggregated.zone,RU - Russia" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sg-aggregated.zone,SG - Singapore" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sy-aggregated.zone,SY - Syria" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/tr-aggregated.zone,TR - Turkey" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ua-aggregated.zone,UA - Ukraine" "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/vn-aggregated.zone,VN - Vietnam" ) # File name with the current date and time output_file="merged_list_${current_datetime}.txt" # Download and merge the IP lists for url_info in "${urls[@]}" do # Splitting the URL information IFS=',' read -r url description <<< "$url_info" # Add comment with the URL and description echo "# URL: $url" >> "$output_file" echo "# Description: $description" >> "$output_file" echo "Download IP list from $url" # Download the IP list and add it to the file curl -sS "$url" >> "$output_file" done echo "Merge completed! The merged list is stored in $output_file" # Formatting the file for the Cloudron Blocklist API formatted_file="formatted_$output_file" # # Add "\n" to the end of each line awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file" # Cloudron Blocklist API endpoint cloudron_api_endpoint="https://yourcloudrondomain.com/api/v1/network/blocklist" # API Key for authentication (replace 'your-api-key' with your API key) api_key="your-api-key" # Upload to Cloudron Blocklist API with wget echo "# Upload to Cloudron Blocklist API with wget..." # The data in the required format for the API data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}" # Send the file with a POST request via wget echo "$data"> temp_data.txt wget --method=POST --header="Content-Type: application/json" --header="Authorization: Bearer $api_key" --body-file=temp_data.txt "$cloudron_api_endpoint" --quiet --output-document=output.txt # Show the output of wget cat output.txt # Delete temporary files rm temp_data.txt
-
Sorry for the Dutch language in the script, ChatGPT is wonderfully writing Dutch without asking haha.
Strange it doesn't seem to work, I extended the script later to upload it in one run to 3 different Cloudrons and that also works perfect. The part of creating the
temp_data.txt
is exactly the same.- Do you see the prompt "# Upload to Cloudron Blocklist API with wget..."?
- What happens after that prompt is showed?
- It takes with these lists about 1,5 minute (on a AMD EPYC 7702P 64-Core Processor with 32GB memory) to process it via API so you have to have some patience
(This is however exactly the same time like when you use the copy/paste in the GUI).
- After the script is finished the
temp_data.txt
is deleted, did the script ever finished? - I execute the script in a LAMP app (on one of the Cloudrons) with only 256MB memory
- last maybe a stupid question but you've set the API token to "read/write"?
I just executed the script (my 3 Cloudrons version) en after about 5 minutes all 3 Cloudrons had the new lists with 60562 IPs (ranges) blocked.
-
I executed the script with
sh script.sh
instead ofbash script.sh
- after doing the latter it worked!
Thanks again.BTW the
output.txt
only contains{}
...? -
@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:
after doing the latter it worked
aha ok, I always use
./script.sh
indeed
output.txt
also with me contains no data, as I read the script the right way it should contain errors duringwget
-
Super!
I have used this script now viacron
and everything seems to work fine, including a significant reduction of "denied" mail attempts in the mail log.I've added the following lines to keep the last 20 url lists, compressed with 7z (which I prefer for compression), for analysis (if needed):
7z a -mx9 "${current_datetime}.7z" "formatted_$output_file" rm "formatted_$output_file" rm "$output_file" ls -td *.7z | grep -v '/$' | tail -n +20 | while IFS= read -r f; do rm -f "$f"; do>
Also: If you use the script, don't just blindly add url-lists. I already managed to lock myself out once by adding the "standard" Firehol list (https://iplists.firehol.org/files/firehol_level1.netset)
-
N necrevistonnezr referenced this topic on
-
N necrevistonnezr referenced this topic on
-
With @imc67 permission, I created a community guide from this thread...
-
... thought I got a "Failed pipeline for master | 3dbdac62" notification...