Cloudron makes it easy to run web apps like WordPress, Nextcloud, GitLab on your server. Find out more or install now.


  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
Skins
  • Light
  • Cerulean
  • Cosmo
  • Flatly
  • Journal
  • Litera
  • Lumen
  • Lux
  • Materia
  • Minty
  • Morph
  • Pulse
  • Sandstone
  • Simplex
  • Sketchy
  • Spacelab
  • United
  • Yeti
  • Zephyr
  • Dark
  • Cyborg
  • Darkly
  • Quartz
  • Slate
  • Solar
  • Superhero
  • Vapor

  • Default (No Skin)
  • No Skin
Collapse

Cloudron Forum

Apps | Demo | Docs | Install

Script to check nginx access logs for 403, 404 and check IP's with greynoise API then send to cloudflare.

Scheduled Pinned Locked Moved Discuss
12 Posts 5 Posters 420 Views
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • M Offline
    M Offline
    Mastadamus
    wrote on last edited by
    #1

    I wrote a script that can do the following.

    1. Check NGINX access.log for 404 and 403 response codes
    2. Extract IP's associated and see if they are scanners/noise with the community greynoise API.
    3. If they are noise, you can pass a -c flag and send them to a cloudflare IP block list to decrease "noise"
    4. If they aren't classified as noise, it can then search the access logs for all activity by these IPs. It will then give a unique count of each URL attempted to be accessed and the response code so that you can perform long tail analysis.

    You can find it here. Feel free to use/cut/slice tailor to your liking

    https://dev.azure.com/Mastadamus/_git/Cloudflare WAF Scripts?path=/GrabIPandCheck.sh

    micmcM 1 Reply Last reply
    5
  • robiR Offline
    robiR Offline
    robi
    wrote on last edited by
    #2

    Can you modify this for directly adding to the Cloudron IP block list?

    Life of sky tech

    M 1 Reply Last reply
    2
  • M Offline
    M Offline
    Mastadamus
    replied to robi on last edited by
    #3

    @robi let me look into it. Need to see how I would interact with that list.

    fbartelsF 1 Reply Last reply
    2
  • fbartelsF Offline
    fbartelsF Offline
    fbartels App Dev
    replied to Mastadamus on last edited by
    #4

    @Mastadamus slightly off topic, but how does greynoise compare to crowded?

    M 1 Reply Last reply
    0
  • M Offline
    M Offline
    Mastadamus
    replied to fbartels on last edited by
    #5

    @fbartels greynoise just runs a big honeypot and sensor landscape that captures and catalogues internet scanning. whereas crowdsec is a community sourced reporting of both scanning and malicious actions. Greynoise I believe has a much larger dataset. Its primary purpose is to see if an IP is targeting just you or multiple entities.

    fbartelsF 1 Reply Last reply
    1
  • fbartelsF Offline
    fbartelsF Offline
    fbartels App Dev
    replied to Mastadamus on last edited by
    #6

    @Mastadamus ah, very interesting.

    1 Reply Last reply
    0
  • micmcM Offline
    micmcM Offline
    micmc
    replied to Mastadamus on last edited by
    #7

    @Mastadamus Sounds enough amazing to me, and with @robi 's suggestion that would even be greater, now wouldn't a script in the like be possible for Apache as well?

    I mean I run wp plugin on my wp sites that detects all 404 like I've never seen before and many of them are obviously scans trying to find ways to attein some presumably installed backend scripts or even the .env file in directly in documentroot. When I discovered that is where I thought that something in the like of what you described here above, would be very useful. 😊


    https://marketingtechnology.agency
    For cutting edge web technologies

    M 1 Reply Last reply
    0
  • M Offline
    M Offline
    Mastadamus
    replied to micmc on last edited by
    #8

    @micmc Yes you could easily take this script and use it for apache. The only think you need to do is look at your apache logs and see what position the source IP is in. the parts in the script where it does awk '{print $1}' etc. are telling it to grab the first position in the nginx log which happens to be the source IP for my particular logging configuration. so really the only think you would need to tailor/alter is the awk statements.

    micmcM 1 Reply Last reply
    1
  • micmcM Offline
    micmcM Offline
    micmc
    replied to Mastadamus on last edited by
    #9

    @Mastadamus
    Sounds great, will take a closer look then thanks a lot mate.


    https://marketingtechnology.agency
    For cutting edge web technologies

    1 Reply Last reply
    0
  • M Offline
    M Offline
    Mastadamus
    wrote on last edited by
    #10

    @girish If I add IP's to /home/yellowtent/platformdata/firewall/blocklist.txt will the automatically be blocked or will I need to restart the box service?

    girishG 1 Reply Last reply
    0
  • girishG Offline
    girishG Offline
    girish Staff
    replied to Mastadamus on last edited by
    #11

    @Mastadamus systemctl restart cloudron-firewall will read that file and apply the rules.

    (As a warning, this file gets re-written if you go to Network -> Block addresses. So, you might lose your changes)

    M 1 Reply Last reply
    1
  • M Offline
    M Offline
    Mastadamus
    replied to girish on last edited by
    #12

    @girish Gotcha but as long as I don't do that, I should be gtg. Im just thinking of a script that does 3 things.

    1. Grabs all the IP's from emerging threats block list
    2. Grabs all the 403/404's from access logs sends them to greynoise to check if they are known "noise" and then
    3. Add both of these IP groups to that file and restart the service.
    1 Reply Last reply
    3

  • Login

  • Don't have an account? Register

  • Login or register to search.
  • First post
    Last post
0
  • Categories
  • Recent
  • Tags
  • Popular
  • Bookmarks
  • Login

  • Don't have an account? Register

  • Login or register to search.