Proxy Outgoing Requests
-
Hello,
We're facing some issues where it seems that some of our web requests for apps like Miniflux, N8N, changedetection.io, and a few others are starting to get blocked.Does anyone have any recommendations on countering this? For example if we setup NGINX Proxy Manager or some sort of other type of app or service?
The preference would be to use a Cloudron App to make administrative tasks a bit easier, if possible. I'd be open to either an App Store App or pulling a custom docker image to our registry and deploying that on Cloudron.
Also, at this point if I just needed to pay for a squid proxy or another type of external service, that might be worth it at this point. If anyone has any recommendations on that, that would be helpful, at least not having to also worry about rotating IP Addresses or getting "red flagged" by my server provider.
The only other thought I had was running NGINX Proxy Manager or Caddy and configuring it to use Cloudflare Workers as a proxy.
Thank you,
Justin -
how much external data do you query to get blocked?
-
G girish moved this topic from Support on
-
Hello,
We're facing some issues where it seems that some of our web requests for apps like Miniflux, N8N, changedetection.io, and a few others are starting to get blocked.Does anyone have any recommendations on countering this? For example if we setup NGINX Proxy Manager or some sort of other type of app or service?
The preference would be to use a Cloudron App to make administrative tasks a bit easier, if possible. I'd be open to either an App Store App or pulling a custom docker image to our registry and deploying that on Cloudron.
Also, at this point if I just needed to pay for a squid proxy or another type of external service, that might be worth it at this point. If anyone has any recommendations on that, that would be helpful, at least not having to also worry about rotating IP Addresses or getting "red flagged" by my server provider.
The only other thought I had was running NGINX Proxy Manager or Caddy and configuring it to use Cloudflare Workers as a proxy.
Thank you,
Justin@BrutalBirdie They just block by IP and User Agent. It's quite random based on the website.
@JLX89 We faced this as well when setting up tracker for minecraft releases. With some trial and error, we found that one has to set the User Agent headers and Accept headers correctly and it bypasses things.
This is how we set it up:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0 Accept-Language: en,de;q=0.5The upstream project suggests using BrightBox or Oxylabs for this. See https://github.com/dgtlmoon/changedetection.io#proxy-configuration . These services offer Residental, ISP, Rotating IPs... They have a referral link in the README btw.
-
@BrutalBirdie They just block by IP and User Agent. It's quite random based on the website.
@JLX89 We faced this as well when setting up tracker for minecraft releases. With some trial and error, we found that one has to set the User Agent headers and Accept headers correctly and it bypasses things.
This is how we set it up:
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/116.0 Accept-Language: en,de;q=0.5The upstream project suggests using BrightBox or Oxylabs for this. See https://github.com/dgtlmoon/changedetection.io#proxy-configuration . These services offer Residental, ISP, Rotating IPs... They have a referral link in the README btw.
@BrutalBirdie said in Proxy Outgoing Requests:
how much external data do you query to get blocked?
Funny thing, it doesn't seem to be much data -- I mean changedetection.io is one thing, but even certain sites I have in Miniflux, typical amount of data I guess?
@girish Thanks! I'll have to take a look at those services and I'll adjust the user agent for changedetection.io. Any thoughts on some sort of solution to tackle some of the other apps; such as n8n, Miniflux, etc?
Thank you!
-
@BrutalBirdie said in Proxy Outgoing Requests:
how much external data do you query to get blocked?
Funny thing, it doesn't seem to be much data -- I mean changedetection.io is one thing, but even certain sites I have in Miniflux, typical amount of data I guess?
@girish Thanks! I'll have to take a look at those services and I'll adjust the user agent for changedetection.io. Any thoughts on some sort of solution to tackle some of the other apps; such as n8n, Miniflux, etc?
Thank you!
@JLX89 Usually, the http requests have to be configured per app since they use libraries and what not. For miniflux, https://miniflux.app/docs/configuration.html#http-client-proxy . n8n seems to support the pseudo standard http_proxy and https_proxy env vars - https://community.n8n.io/t/http-proxy-problem/8787/
-
Interestingly , miniflux has this change - https://github.com/miniflux/v2/commit/46c13b51851374b445ed0458c321808bd83e8869 . which per https://pkg.go.dev/net/http#ProxyFromEnvironment means that setting those env vars will work on miniflux as well probably.
-
@JLX89 Usually, the http requests have to be configured per app since they use libraries and what not. For miniflux, https://miniflux.app/docs/configuration.html#http-client-proxy . n8n seems to support the pseudo standard http_proxy and https_proxy env vars - https://community.n8n.io/t/http-proxy-problem/8787/
-
@girish Perfect, thanks! I'll have to take a look around and see if there is an or something I can setup on Cloudron for a proxy.
@JLX89 So, the important thing is the IP and User Agent. It needs to appear as if it's coming from a business or residence or data center as opposed to some cloud server (bots). Maybe you can try setting up squid/varnish (the heavy weights...)?
-
I'm facing blocking IP address problem with Changedetection.io. I tried to use BrightBox or Oxylabs but it seems those service are not working correctly, specially with some kind of websites that require steps actions to perform.
Do you have any suggestion how I can rotate IP addresses on a Cloudron Changedetection.io instance?
