How to proctect instances from Bot, Crawlers, Requests, & Co?

[fbartels]

If the bots are compliant to it https://en.wikipedia.org/wiki/Robots.txt would be the tool you are looking for. This file can already be managed through the Cloudron UI.

When it comes to preventing bad actors then https://docs.crowdsec.net/ could be worthwhile to look into.

[p44]

@fbartels Yes, Robots.txt, .htaccess, all good... but it could be great to manage rules in a central (and simple) way, special on Cloudron instances with multiple apps installed.

It seems to be little bit complicated for my skills. I had a look on this post.

Are you using Crowdsec?

[jdaviescoates]

I always install Wordfence on all my WordPress sites. Blocks most stuff.

[joseph]

Is there a service out there that provides AI block lists? crowdsec only has a platinum list at 3500/month ?

[p44]

@jdaviescoates @joseph Thanks, I wouldn't want to rely on outside services.

[jdaviescoates]

@p44 TBH wrt WordPress i'd expect Wordfence would likely do a much better job that you'd ever be able to do manually. They have very long blocklists and know about many more bad IPs than you do. I just use the free version.

[p44]

@jdaviescoates Thanks, it seems that they don’t block AI bots...

[robi]

Can you extract the blocklists from Wordfence to have it populate the Cloudron deny list?

[p44]

@robi Thanks for advice.

I don’t know where and if Wordfence has a public list, but I think that blocklists has a lot of data that can be huge to handle from CPU.

[robi]

@p44 those are solved problems with lookup tables.

[jdaviescoates]

@p44 I think Wordfence adds stuff to .htaccess

[p44]

@robi Thanks a lot.

I found and applied specific rules in Wordpress .htaccess:

• 8G Firewall
• Ultimate Block List to Stop AI Bots

I think is a good start.

Both filters it seems working fine. Of course, it would better to manage and deploy centrally.

Thanks again for your advices Robi.

[robi]

@p44 You are very welcome.

Now making a tool to parse those IPs for the Cloudron block list is something an eager LLM agent could do.

[p44]

@robi Amazing!!!

Something like a “bridge” that every day take from the source and update Cloudron block list...