Bloquejar crawlers agressius i bots#

Al fitxer .htaccess afegir la següent configuració:

# blocking aggresive crawlers and bots
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]

També es pot optar per:

BrowserMatchNoCase Linguee Bot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

En un worpress, el fitxer .htaccess quedaria:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
# blocking aggresive crawlers and bots https://www.johnlarge.co.uk/blocking-aggressive-chinese-crawlers-scrapers-bots/
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]

RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
BrowserMatchNoCase Linguee Bot bad_bot
BrowserMatchNoCase SemrushBot bad_bot
Order Deny,Allow
Deny from env=bad_bot

Font johnlarge.co.uk