title: fail2ban force me to ban google because of /forward in my log
body: <p>In my apache log, I have a lot of stuff like this:</p>
  
  <pre><code>&lt;IP ADDRESS&gt; - - &lt;DATE&gt; "GET /forward?path=http://vary_bad_link_not_for_children" &lt;NUM1&gt; &lt;NUM2&gt; "-" &lt;String&gt;
  </code></pre>
  
  <p>&lt;NUM1&gt;: 302 or 404</p>
  
  <p>&lt;NUM2&gt;: 5XX, 6XX or 11XX</p>
  
  <p>&lt;String&gt;: </p>
  
  <p>"Mozilla/5.0 (compatible; AhrefsBot/5.1; +<a href="http://ahrefs.com/robot/" rel="nofollow">http://ahrefs.com/robot/</a>)"</p>
  
  <p>"Mozilla/5.0 (compatible; MJ12bot/v1.4.5; <a href="http://www.majestic12.co.uk/bot.php?+" rel="nofollow">http://www.majestic12.co.uk/bot.php?+</a>)"</p>
  
  <p>"Mozilla/5.0 (compatible; Googlebot/2.1; +...a link)"</p>
  
  <p>"Mozilla/5.0 (compatible; Exabot/3.0; +...a link)"</p>
  
  <p>etc...</p>
  
  <p>I have made a jail for fail2ban with this regex:</p>
  
  <pre><code>failregex = ^&lt;HOST&gt; .*"GET .*/forward\?path=
  </code></pre>
  
  <p>Everything is working fine except that the IP address that are banned (see &lt;IP ADDRESS&gt; in the log) are the IP of google and other very well known companies.</p>
  
  <p>I really don't understand why it is like this; I mean why should I ban google and the other companies and If not, Why should I accept all those inappropriate request to my server.</p>
  
  <p>I would like to clarify my questions, as it was poorly explained:</p>
  
  <p>1-Why Google IP (and other known companies) are doing those kind of "porn" requests</p>
  
  <p>2-Is there any meaning to "/forward?path=..." is it an apache feature?</p>
  
  <p>3-How to handle this problem without stopping the "good" bots to reference my sites.</p>
  
  <p>Thanks by advance for any help!</p>