Script to check nginx access logs for 403, 404 and check IP's with greynoise API then send to cloudflare.

[Mastadamus]

I wrote a script that can do the following.

1. Check NGINX access.log for 404 and 403 response codes
2. Extract IP's associated and see if they are scanners/noise with the community greynoise API.
3. If they are noise, you can pass a -c flag and send them to a cloudflare IP block list to decrease "noise"
4. If they aren't classified as noise, it can then search the access logs for all activity by these IPs. It will then give a unique count of each URL attempted to be accessed and the response code so that you can perform long tail analysis.

You can find it here. Feel free to use/cut/slice tailor to your liking

https://dev.azure.com/Mastadamus/_git/Cloudflare WAF Scripts?path=/GrabIPandCheck.sh

[robi]

Can you modify this for directly adding to the Cloudron IP block list?

[Mastadamus]

@robi let me look into it. Need to see how I would interact with that list.

[fbartels]

@Mastadamus slightly off topic, but how does greynoise compare to crowded?

[Mastadamus]

@fbartels greynoise just runs a big honeypot and sensor landscape that captures and catalogues internet scanning. whereas crowdsec is a community sourced reporting of both scanning and malicious actions. Greynoise I believe has a much larger dataset. Its primary purpose is to see if an IP is targeting just you or multiple entities.

[fbartels]

@Mastadamus ah, very interesting.

[micmc]

@Mastadamus Sounds enough amazing to me, and with @robi 's suggestion that would even be greater, now wouldn't a script in the like be possible for Apache as well?

I mean I run wp plugin on my wp sites that detects all 404 like I've never seen before and many of them are obviously scans trying to find ways to attein some presumably installed backend scripts or even the .env file in directly in documentroot. When I discovered that is where I thought that something in the like of what you described here above, would be very useful.

[Mastadamus]

@micmc Yes you could easily take this script and use it for apache. The only think you need to do is look at your apache logs and see what position the source IP is in. the parts in the script where it does awk '{print $1}' etc. are telling it to grab the first position in the nginx log which happens to be the source IP for my particular logging configuration. so really the only think you would need to tailor/alter is the awk statements.

[micmc]

@Mastadamus
Sounds great, will take a closer look then thanks a lot mate.

[Mastadamus]

@girish If I add IP's to /home/yellowtent/platformdata/firewall/blocklist.txt will the automatically be blocked or will I need to restart the box service?

[girish]

@Mastadamus systemctl restart cloudron-firewall will read that file and apply the rules.

(As a warning, this file gets re-written if you go to Network -> Block addresses. So, you might lose your changes)

[Mastadamus]

@girish Gotcha but as long as I don't do that, I should be gtg. Im just thinking of a script that does 3 things.

1. Grabs all the IP's from emerging threats block list
2. Grabs all the 403/404's from access logs sends them to greynoise to check if they are known "noise" and then
3. Add both of these IP groups to that file and restart the service.