<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Is there a way to rate limit connections to a site for certain user agent strings?]]></title><description><![CDATA[<p dir="auto">Hello,</p>
<p dir="auto">I have a particular website that for the last 2+ days has been reaching max memory and restarting frequently, a dozen times a day. I've tried increasing the memory which has helped of course but that's only a temporary workaround. The issue started when (according to the logs) the site started receiving an onslaught of traffic from Facebook crawler bots, specifically their <code>Meta-ExternalAgent/1.1</code> one.</p>
<p dir="auto">What I'd like to do is try to rate limit (within Cloudron if possible) the requests from certain user agents, to maybe 10 a minute for example instead of several a second (which is currently what I'm seeing). If this is possible, I'd love to know.</p>
<p dir="auto">I may be able to use a plugin in WordPress to do that but my thinking is this will still take up Apache connections which can still saturate the connections. In fact I tried to do this with the .htaccess using something ChatGPT recommended, but this just slows down the data rate and doesn't really slow down the indexing from Facebook / Meta, so I suspect this will simply increase the connection saturation if each request takes a bit longer to respond to.</p>
<pre><code># BEGIN Meta-ExternalHit Throttling
&lt;IfModule mod_rewrite.c&gt;
    RewriteEngine On
    # Detect Meta-ExternalHit user agent
    RewriteCond %{HTTP_USER_AGENT} "Meta-ExternalHit" [NC]
    # Set an env var if matched
    RewriteRule ^ - [E=IS_META_BOT:1]
&lt;/IfModule&gt;

&lt;IfModule mod_ratelimit.c&gt;
    # Apply rate limit if Meta bot detected
    SetEnvIf IS_META_BOT 1 META_BOT
    &lt;IfModule mod_filter.c&gt;
        AddOutputFilterByType RATE_LIMIT text/html text/plain text/xml application/json application/xml image/jpeg image/png image/webp image/avif
    &lt;/IfModule&gt;
    # Limit to ~50 KB/s (value is KB per second)
    SetEnvIf META_BOT 1 RATE_LIMIT 50
&lt;/IfModule&gt;
# END Meta-ExternalHit Throttling
</code></pre>
<p dir="auto">This is leading to the health checks taking over 7000ms as well which I see in the logs.</p>
<p dir="auto">Thank you in advance for any advice.</p>
]]></description><link>https://forum.cloudron.io/topic/14343/is-there-a-way-to-rate-limit-connections-to-a-site-for-certain-user-agent-strings</link><generator>RSS for Node</generator><lastBuildDate>Mon, 20 Jul 2026 09:43:08 GMT</lastBuildDate><atom:link href="https://forum.cloudron.io/topic/14343.rss" rel="self" type="application/rss+xml"/><pubDate>Wed, 01 Oct 2025 06:08:51 GMT</pubDate><ttl>60</ttl><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Sat, 31 Jan 2026 12:50:12 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> the past 2 days I rented a VPS, installed Bunkerweb and played with it but its way to much and way too complicated! I created this <a href="https://forum.cloudron.io/topic/14982/feature-request-simple-per-app-waf-with-templates-kiss">https://forum.cloudron.io/topic/14982/feature-request-simple-per-app-waf-with-templates-kiss</a> and hope the <a class="plugin-mentions-user plugin-mentions-a" href="/user/girish" aria-label="Profile: girish">@<bdi>girish</bdi></a> and <a class="plugin-mentions-user plugin-mentions-a" href="/user/nebulon" aria-label="Profile: nebulon">@<bdi>nebulon</bdi></a> will have a look at it?</p>
]]></description><link>https://forum.cloudron.io/post/119444</link><guid isPermaLink="true">https://forum.cloudron.io/post/119444</guid><dc:creator><![CDATA[imc67]]></dc:creator><pubDate>Sat, 31 Jan 2026 12:50:12 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Tue, 27 Jan 2026 18:55:51 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/imc67" aria-label="Profile: imc67">@<bdi>imc67</bdi></a> Sure, but not all apps need it.<br />
WP exploits do not work on surfer or cubby or ghost.</p>
<p dir="auto">It does seem like this is a gap in the market though, webservers accepting and rejecting their own traffic. Filter where you need it.</p>
<p dir="auto">They have to process all the headers anyway.</p>
]]></description><link>https://forum.cloudron.io/post/119256</link><guid isPermaLink="true">https://forum.cloudron.io/post/119256</guid><dc:creator><![CDATA[robi]]></dc:creator><pubDate>Tue, 27 Jan 2026 18:55:51 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Tue, 27 Jan 2026 18:50:37 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/robi" aria-label="Profile: robi">@<bdi>robi</bdi></a> said in <a href="/post/119251">Is there a way to rate limit connections to a site for certain user agent strings?</a>:</p>
<blockquote>
<p dir="auto">Curious, why isn't an app level WAF like the ones for WP suitable?</p>
</blockquote>
<p dir="auto">Then you have to find for each app (not only WP) you want to protect a specific WAF plugin?!</p>
]]></description><link>https://forum.cloudron.io/post/119255</link><guid isPermaLink="true">https://forum.cloudron.io/post/119255</guid><dc:creator><![CDATA[imc67]]></dc:creator><pubDate>Tue, 27 Jan 2026 18:50:37 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Tue, 27 Jan 2026 17:28:57 GMT]]></title><description><![CDATA[<p dir="auto">Curious, why isn't an app level WAF like the ones for WP suitable?</p>
]]></description><link>https://forum.cloudron.io/post/119251</link><guid isPermaLink="true">https://forum.cloudron.io/post/119251</guid><dc:creator><![CDATA[robi]]></dc:creator><pubDate>Tue, 27 Jan 2026 17:28:57 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Tue, 27 Jan 2026 14:35:21 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/imc67" aria-label="Profile: imc67">@<bdi>imc67</bdi></a> <a href="https://bunny.net/blog/bunny-shield-waf-fast-flexible-and-regex-ready/" target="_blank" rel="noopener noreferrer nofollow ugc">https://bunny.net/blog/bunny-shield-waf-fast-flexible-and-regex-ready/</a></p>
<p dir="auto"><a href="https://bunny.net/shield/" target="_blank" rel="noopener noreferrer nofollow ugc">https://bunny.net/shield/</a></p>
]]></description><link>https://forum.cloudron.io/post/119240</link><guid isPermaLink="true">https://forum.cloudron.io/post/119240</guid><dc:creator><![CDATA[humpty]]></dc:creator><pubDate>Tue, 27 Jan 2026 14:35:21 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Tue, 27 Jan 2026 14:20:16 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> thanks for the very detailed experiences up to now! I was searching (partly via AI) the web for Cloudflare WAF alternatives and it's really unbelievable they are sooo rare!</p>
<p dir="auto">As long as Cloudron doesn't have anything like WAF (on app and URL-parts) and one from Europe wants to leave Cloudflare there is not much choice <img src="https://forum.cloudron.io/assets/plugins/nodebb-plugin-emoji/emoji/android/1f62b.png?v=af5271e93de" class="not-responsive emoji emoji-android emoji--tired_face" style="height:23px;width:auto;vertical-align:middle" title="😫" alt="😫" /></p>
]]></description><link>https://forum.cloudron.io/post/119239</link><guid isPermaLink="true">https://forum.cloudron.io/post/119239</guid><dc:creator><![CDATA[imc67]]></dc:creator><pubDate>Tue, 27 Jan 2026 14:20:16 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Mon, 26 Jan 2026 22:54:17 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/imc67" aria-label="Profile: imc67">@<bdi>imc67</bdi></a> I never found the time to delve deeper into the test system (WAF – here Bunkerweb – in front of a dedicated Cloudron instance). Conversely: I completely missed my own challenge months ago. Thanks for bringing that up again. <img src="https://forum.cloudron.io/assets/plugins/nodebb-plugin-emoji/emoji/android/1f642.png?v=af5271e93de" class="not-responsive emoji emoji-android emoji--slightly_smiling_face" style="height:23px;width:auto;vertical-align:middle" title=":)" alt="🙂" /></p>
<p dir="auto">With Bunker we're on the free tier. One thing missing in the free tier is: reporting/monitoring over a longer period. So no direct insight into the numbers the WAF filters out. But from our experience with one app on Cloudron (our own website): no downtime, no stress, nothing. Everything as expected after some manual configurations.</p>
<ol>
<li>
<p dir="auto">From a marketing perspective: Filtering out bots causes problems. No link previews in LinkedIn, Rocket.Chat, Signal... Problem solved by allowlists for some User-Agents. But in the long run it feels wrong to only pay attention to User-Agents. Bad bots find solutions to adopt the "good" User-Agents. In that case I don't think the WAF will work. We'll see.</p>
</li>
<li>
<p dir="auto">Our website runs on Drupal. We added custom rules to forbid certain URL structures. What we learned: Some editors use workflows that generate URL structures that were forbidden. So they asked the Bunker administrators to change the structure to enable their work.</p>
</li>
<li>
<p dir="auto">Our first update of Bunkerweb ended directly in a disaster. The maintainers rolled back the update, we reverted the version and a few days later a new update was released. That works. The last two updates worked without problems.</p>
</li>
<li>
<p dir="auto">Is the time investment worth it? I think so. We have so few answers on alternatives to Cloudflare. We need a solid free and open-source alternative. What we learned: When it works, it works. You have to learn some new terminologies and technologies. It makes us stronger in decisions and better at consulting. Is it as good as Cloudflare? Maybe later, from my point of view. Don't forget that an important issue is DOS attacks. This is not solved with Bunkerweb in the free version on a Hetzner VPS.</p>
</li>
</ol>
<p dir="auto">Once I find time to dedicate myself to the test system (Cloudron instance behind Bunkerweb) again, I will post an update. Many thanks for the reminder.</p>
]]></description><link>https://forum.cloudron.io/post/119192</link><guid isPermaLink="true">https://forum.cloudron.io/post/119192</guid><dc:creator><![CDATA[luckow]]></dc:creator><pubDate>Mon, 26 Jan 2026 22:54:17 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Mon, 26 Jan 2026 14:44:03 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> said in <a href="/post/113275">Is there a way to rate limit connections to a site for certain user agent strings?</a>:</p>
<blockquote>
<p dir="auto">Bunkerweb acts as a reverse proxy for a Cloudron app that is ‘behind it’. Currently, we only use it in front of our own website (mainly because we are still learning, e.g. what happens when we block bots? Oh, there is no longer support for previews in rocket.chat). In my next spare moment, I'll try out what happens when a complete Cloudron instance is behind Bunkerweb. It should work. From what I've heard, this is the case with Cloudflare, and Bunkerweb is similar (only self-hosted) <img src="https://forum.cloudron.io/assets/plugins/nodebb-plugin-emoji/emoji/android/1f642.png?v=af5271e93de" class="not-responsive emoji emoji-android emoji--slightly_smiling_face" style="height:23px;width:auto;vertical-align:middle" title=":)" alt="🙂" /></p>
</blockquote>
<p dir="auto">Hi <a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> I'm really curious how it went with Bunkerweb in front of Cloudron?</p>
<p dir="auto">I am moving domains from Cloudflare to deSEC but can't do all because I use Cloudflare WAF for some Cloudron-apps (Geoblocking and/or IP whitelist with DDNS/API on app-level). And because Cloudron doesn't have anything like a WAF a workaround (what a pity) could be Bunkerweb?</p>
]]></description><link>https://forum.cloudron.io/post/119170</link><guid isPermaLink="true">https://forum.cloudron.io/post/119170</guid><dc:creator><![CDATA[imc67]]></dc:creator><pubDate>Mon, 26 Jan 2026 14:44:03 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Thu, 02 Oct 2025 05:00:26 GMT]]></title><description><![CDATA[<p dir="auto">Thank you all for the suggestions! Good ideas!</p>
<p dir="auto">Overnight after my message to the forum it turns out the bot traffic finally went back to normal levels overnight and the app has been stable ever since. But this definitely reminded me that getting a good WAF (or improving the robots.txt at a minimum) can be important and needs to be evaluated.</p>
<p dir="auto">Hopefully Cloudron can integrate a simplistic WAF into the system directly in the future (maybe even using that BunkerWeb if possible). <img src="https://forum.cloudron.io/assets/plugins/nodebb-plugin-emoji/emoji/android/1f91e.png?v=af5271e93de" class="not-responsive emoji emoji-android emoji--hand_with_index_and_middle_fingers_crossed" style="height:23px;width:auto;vertical-align:middle" title=":hand_with_index_and_middle_fingers_crossed:" alt="🤞" /></p>
]]></description><link>https://forum.cloudron.io/post/113283</link><guid isPermaLink="true">https://forum.cloudron.io/post/113283</guid><dc:creator><![CDATA[d19dotca]]></dc:creator><pubDate>Thu, 02 Oct 2025 05:00:26 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 17:26:55 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/d19dotca" aria-label="Profile: d19dotca">@<bdi>d19dotca</bdi></a> Think what you want of Cloudflare but their caching is prettyy good, plus they also hate AI Bots and have specific options to block them: <a href="https://developers.cloudflare.com/ai-crawl-control/" target="_blank" rel="noopener noreferrer nofollow ugc">https://developers.cloudflare.com/ai-crawl-control/</a></p>
]]></description><link>https://forum.cloudron.io/post/113276</link><guid isPermaLink="true">https://forum.cloudron.io/post/113276</guid><dc:creator><![CDATA[andreasdueren]]></dc:creator><pubDate>Wed, 01 Oct 2025 17:26:55 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 16:46:07 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/jdaviescoates" aria-label="Profile: jdaviescoates">@<bdi>jdaviescoates</bdi></a><br />
The good old traditional method: <a href="https://docs.bunkerweb.io/latest/integrations/#linux" target="_blank" rel="noopener noreferrer nofollow ugc">https://docs.bunkerweb.io/latest/integrations/#linux</a><br />
Runs on a CX22 on <a href="https://www.hetzner.com/cloud/" target="_blank" rel="noopener noreferrer nofollow ugc">https://www.hetzner.com/cloud/</a>.</p>
<p dir="auto">Bunkerweb acts as a reverse proxy for a Cloudron app that is ‘behind it’. Currently, we only use it in front of our own website (mainly because we are still learning, e.g. what happens when we block bots? Oh, there is no longer support for previews in rocket.chat). In my next spare moment, I'll try out what happens when a complete Cloudron instance is behind Bunkerweb. It should work. From what I've heard, this is the case with Cloudflare, and Bunkerweb is similar (only self-hosted) <img src="https://forum.cloudron.io/assets/plugins/nodebb-plugin-emoji/emoji/android/1f642.png?v=af5271e93de" class="not-responsive emoji emoji-android emoji--slightly_smiling_face" style="height:23px;width:auto;vertical-align:middle" title=":)" alt="🙂" /></p>
]]></description><link>https://forum.cloudron.io/post/113275</link><guid isPermaLink="true">https://forum.cloudron.io/post/113275</guid><dc:creator><![CDATA[luckow]]></dc:creator><pubDate>Wed, 01 Oct 2025 16:46:07 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 16:27:36 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> said in <a href="/post/113228">Is there a way to rate limit connections to a site for certain user agent strings?</a>:</p>
<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/d19dotca" aria-label="Profile: d19dotca">@<bdi>d19dotca</bdi></a> Install your own WAF. We have been testing <a href="https://www.bunkerweb.io/" target="_blank" rel="noopener noreferrer nofollow ugc">https://www.bunkerweb.io/</a> for almost a month. And it works.</p>
</blockquote>
<p dir="auto">Sounds good. How?</p>
]]></description><link>https://forum.cloudron.io/post/113272</link><guid isPermaLink="true">https://forum.cloudron.io/post/113272</guid><dc:creator><![CDATA[jdaviescoates]]></dc:creator><pubDate>Wed, 01 Oct 2025 16:27:36 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 16:24:36 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/luckow" aria-label="Profile: luckow">@<bdi>luckow</bdi></a> said in <a href="/post/113228">Is there a way to rate limit connections to a site for certain user agent strings?</a>:</p>
<blockquote>
<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/d19dotca" aria-label="Profile: d19dotca">@<bdi>d19dotca</bdi></a> Install your own WAF. We have been testing <a href="https://www.bunkerweb.io/" target="_blank" rel="noopener noreferrer nofollow ugc">https://www.bunkerweb.io/</a> for almost a month. And it works.</p>
</blockquote>
<p dir="auto">Would that be interesting as a Cloudron service?</p>
]]></description><link>https://forum.cloudron.io/post/113271</link><guid isPermaLink="true">https://forum.cloudron.io/post/113271</guid><dc:creator><![CDATA[necrevistonnezr]]></dc:creator><pubDate>Wed, 01 Oct 2025 16:24:36 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 14:34:33 GMT]]></title><description><![CDATA[<p dir="auto">Another thought it to inspect their robots.txt for any directives for their bots which you may adapt for your needs.</p>
]]></description><link>https://forum.cloudron.io/post/113255</link><guid isPermaLink="true">https://forum.cloudron.io/post/113255</guid><dc:creator><![CDATA[robi]]></dc:creator><pubDate>Wed, 01 Oct 2025 14:34:33 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 09:04:02 GMT]]></title><description><![CDATA[<p dir="auto"><a class="plugin-mentions-user plugin-mentions-a" href="/user/d19dotca" aria-label="Profile: d19dotca">@<bdi>d19dotca</bdi></a> Install your own WAF. We have been testing <a href="https://www.bunkerweb.io/" target="_blank" rel="noopener noreferrer nofollow ugc">https://www.bunkerweb.io/</a> for almost a month. And it works.</p>
]]></description><link>https://forum.cloudron.io/post/113228</link><guid isPermaLink="true">https://forum.cloudron.io/post/113228</guid><dc:creator><![CDATA[luckow]]></dc:creator><pubDate>Wed, 01 Oct 2025 09:04:02 GMT</pubDate></item><item><title><![CDATA[Reply to Is there a way to rate limit connections to a site for certain user agent strings? on Wed, 01 Oct 2025 07:13:19 GMT]]></title><description><![CDATA[<p dir="auto">Anf of course, Meta ignores your <code>robots.txt</code>? It‘s such a sh*t company…</p>
<p dir="auto">Just FYI: If you were in the EU and/or Germany:</p>
<ul>
<li>
<p dir="auto">You can legally prevent AI companies from using your website's data. The legal basis is the <strong>right to opt-out</strong> of Text and Data Mining (TDM) under <a href="https://eur-lex.europa.eu/legal-content/EN/TXT/HTML/?uri=CELEX:32019L0790#art_4" target="_blank" rel="noopener noreferrer nofollow ugc">Art. 4 of the EU Copyright Directive</a>.</p>
</li>
<li>
<p dir="auto">Germany has implemented the directive in <a href="https://www.gesetze-im-internet.de/englisch_urhg/englisch_urhg.html#p0328" target="_blank" rel="noopener noreferrer nofollow ugc">Art. 44b UrhG</a>: „<em>Uses in accordance with subsection (2) sentence 1 are permitted only if they have not been reserved by the rightholder. A reservation of use in the case of works which are available online is effective only if it is made in a machine-readable format.</em>“</p>
</li>
<li>
<p dir="auto">Your objection must be <em>machine-readable</em>. A simple text disclaimer on your site (e.g., in the legal notice) is legally insufficient.</p>
</li>
<li>
<p dir="auto">The standard method is to use the above mentioned <code>robots.txt</code>.</p>
</li>
<li>
<p dir="auto">A comprehensive, community-maintained list can be found at projects like <a href="https://github.com/ai-robots-txt/ai.robots.txt" target="_blank" rel="noopener noreferrer nofollow ugc">ai.robots.txt on GitHub</a>.</p>
</li>
<li>
<p dir="auto">While major companies respect <code>robots.txt</code>, compliance is not guaranteed from all crawlers, but it is the recognized legal and technical standard for opting out.</p>
</li>
</ul>
]]></description><link>https://forum.cloudron.io/post/113211</link><guid isPermaLink="true">https://forum.cloudron.io/post/113211</guid><dc:creator><![CDATA[necrevistonnezr]]></dc:creator><pubDate>Wed, 01 Oct 2025 07:13:19 GMT</pubDate></item></channel></rss>