Firewall / Spamassassin: Automatic list update

[necrevistonnezr]

Thanks again!
I just tried this - it generated merged_list_20240110_091244.txt and formatted_merged_list_20240110_091244.txt but somehow, the temp_data.txt is not generated, and therefore the upload is not successful?

BTW, here's your script with english comments and filenames:

#!/bin/bash

# Current date and time
current_datetime=$(date +"%Y%m%d_%H%M%S")

# Array containing the URLs of the IP lists and their descriptions
declare -a urls=(
	"https://iplists.firehol.org/files/spamhaus_drop.netset,Spamhaus - Drop"
        "https://iplists.firehol.org/files/spamhaus_edrop.netset,Spamhaus - eDrop"
        "https://www.ipdeny.com/ipblocks/data/aggregated/af-aggregated.zone,AF - Afganistan"
        "https://www.ipdeny.com/ipblocks/data/aggregated/bd-aggregated.zone,BD - Bangladesh"
        "https://www.ipdeny.com/ipblocks/data/aggregated/by-aggregated.zone,BY - Belarus"
        "https://www.ipdeny.com/ipblocks/data/aggregated/br-aggregated.zone,BR - Brazil"
        "https://www.ipdeny.com/ipblocks/data/aggregated/cn-aggregated.zone,CN - China"
        "https://www.ipdeny.com/ipblocks/data/aggregated/ir-aggregated.zone,IR - Iran"
        "https://www.ipdeny.com/ipblocks/data/aggregated/in-aggregated.zone,IN - India"
        "https://www.ipdeny.com/ipblocks/data/aggregated/kp-aggregated.zone,KP - North Korea"
        "https://www.ipdeny.com/ipblocks/data/aggregated/np-aggregated.zone,NP - Nepal"
        "https://www.ipdeny.com/ipblocks/data/aggregated/pk-aggregated.zone,PK - Pakistan"
        "https://www.ipdeny.com/ipblocks/data/aggregated/ro-aggregated.zone,RO - Romania"
        "https://www.ipdeny.com/ipblocks/data/aggregated/ru-aggregated.zone,RU - Russia"
        "https://www.ipdeny.com/ipblocks/data/aggregated/sg-aggregated.zone,SG - Singapore"
        "https://www.ipdeny.com/ipblocks/data/aggregated/sy-aggregated.zone,SY - Syria"
        "https://www.ipdeny.com/ipblocks/data/aggregated/tr-aggregated.zone,TR - Turkey"
        "https://www.ipdeny.com/ipblocks/data/aggregated/ua-aggregated.zone,UA - Ukraine"
        "https://www.ipdeny.com/ipblocks/data/aggregated/vn-aggregated.zone,VN - Vietnam"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/af-aggregated.zone,AF - Afganistan"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/bd-aggregated.zone,BD - Bangladesh"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/by-aggregated.zone,BY - Belarus"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/br-aggregated.zone,BR - Brazil"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/cn-aggregated.zone,CN - China"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ir-aggregated.zone,IR - Iran"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/in-aggregated.zone,IN - India"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/np-aggregated.zone,NP - Nepal"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/pk-aggregated.zone,PK - Pakistan"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ro-aggregated.zone,RO - Romania"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ru-aggregated.zone,RU - Russia"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sg-aggregated.zone,SG - Singapore"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/sy-aggregated.zone,SY - Syria"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/tr-aggregated.zone,TR - Turkey"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/ua-aggregated.zone,UA - Ukraine"
        "https://www.ipdeny.com/ipv6/ipaddresses/aggregated/vn-aggregated.zone,VN - Vietnam"
)

# File name with the current date and time
output_file="merged_list_${current_datetime}.txt"

# Download and merge the IP lists
for url_info in "${urls[@]}"
do
    # Splitting the URL information
    IFS=',' read -r url description <<< "$url_info"

    # Add comment with the URL and description
    echo "# URL: $url" >> "$output_file"
    echo "# Description: $description" >> "$output_file"

    echo "Download IP list from $url"
    # Download the IP list and add it to the file
    curl -sS "$url" >> "$output_file"
done

echo "Merge completed! The merged list is stored in $output_file"

# Formatting the file for the Cloudron Blocklist API
formatted_file="formatted_$output_file"

# # Add "\n" to the end of each line
awk '{printf "%s\\n",$0}' "$output_file" > "$formatted_file"

# Cloudron Blocklist API endpoint
cloudron_api_endpoint="https://yourcloudrondomain.com/api/v1/network/blocklist"

# API Key for authentication (replace 'your-api-key' with your API key)
api_key="your-api-key"

# Upload to Cloudron Blocklist API with wget
echo "# Upload to Cloudron Blocklist API with wget..."

# The data in the required format for the API
data="{\"blocklist\":\"$(cat "$formatted_file" | tr '\n' '\\n')\"}"

# Send the file with a POST request via wget
echo "$data"> temp_data.txt
wget --method=POST --header="Content-Type: application/json" --header="Authorization: Bearer $api_key" --body-file=temp_data.txt "$cloudron_api_endpoint" --quiet --output-document=output.txt

# Show the output of wget
cat output.txt

# Delete temporary files
rm temp_data.txt

[imc67]

Sorry for the Dutch language in the script, ChatGPT is wonderfully writing Dutch without asking haha.

Strange it doesn't seem to work, I extended the script later to upload it in one run to 3 different Cloudrons and that also works perfect. The part of creating the temp_data.txt is exactly the same.

• Do you see the prompt "# Upload to Cloudron Blocklist API with wget..."?
• What happens after that prompt is showed?
• It takes with these lists about 1,5 minute (on a AMD EPYC 7702P 64-Core Processor with 32GB memory) to process it via API so you have to have some patience (This is however exactly the same time like when you use the copy/paste in the GUI).
• After the script is finished the temp_data.txt is deleted, did the script ever finished?
• I execute the script in a LAMP app (on one of the Cloudrons) with only 256MB memory
• last maybe a stupid question but you've set the API token to "read/write"?

I just executed the script (my 3 Cloudrons version) en after about 5 minutes all 3 Cloudrons had the new lists with 60562 IPs (ranges) blocked.

[necrevistonnezr]

I executed the script with sh script.sh instead of bash script.sh - after doing the latter it worked!
Thanks again.

BTW the output.txt only contains {}...?

[imc67]

@necrevistonnezr said in Firewall / Spamassassin: Automatic list update:

after doing the latter it worked

aha ok, I always use ./script.sh

indeed output.txt also with me contains no data, as I read the script the right way it should contain errors during wget

[necrevistonnezr]

Super!
I have used this script now via cron and everything seems to work fine, including a significant reduction of "denied" mail attempts in the mail log.

I've added the following lines to keep the last 20 url lists, compressed with 7z (which I prefer for compression), for analysis (if needed):

7z a -mx9 "${current_datetime}.7z" "formatted_$output_file"
rm "formatted_$output_file"
rm "$output_file"
ls -td *.7z | grep -v '/$' | tail -n +20 | while IFS= read -r f; do rm -f "$f"; do>

Also: If you use the script, don't just blindly add url-lists. I already managed to lock myself out once by adding the "standard" Firehol list (https://iplists.firehol.org/files/firehol_level1.netset)

[necrevistonnezr]

With @imc67 permission, I created a community guide from this thread...

[necrevistonnezr]

... thought I got a "Failed pipeline for master | 3dbdac62" notification...

[humptydumpty]

@necrevistonnezr link to the guide, pretty please!

[necrevistonnezr]

https://docs.cloudron.io/guides/community/

[humptydumpty]

@necrevistonnezr I was looking in the forum Thank you!