ai.robots.txt for apache2
=========================

Bots are reported to now make up half of the web's traffic. There are several
ways of blocking these.

While there are more advanced tools like Anubis[0], for smaller sites often the
simpler approach of blocking access by user-agent is currently still enough.

Here are the different ways how to enable this blocklist with your website.


1. Blocking bots from within the apache2 configuration
------------------------------------------------------

Preferably if you have write access to the apache2 configuration,
the ai-bot configuration snipped can be included in individual VirtualHost directives:

---snip---
<VirtualHost www.example.org>
  Include /etc/apache2/conf-available/block-ai-bots.conf
  [...]

</VirtualHost>
---snap---

or be enabled globally:

  $ sudo a2enconf block-ai-bots
  $ sudo service apache2 reload


2. Blocking bots from within the websites document root
-------------------------------------------------------

Alternatively you can symlink the htaccess file into your document root, e.g.:

  $ ln -s /usr/share/apache2-ai-bots/htaccess /var/www/html/.htaccess
  $ ln -s /usr/share/apache2-ai-bots/robots.txt /var/www/html/robots.txt

Please make sure that htaccess usage is enabled in your apache2 configuration
(AllowOverride setting) and that symlinks are followed
(Options +SymLinksIfOwnerMatch or +FollowSymLinks, depending on your setup).


[0] https://github.com/TecharoHQ/anubis
    https://bugs.debian.org/1102132

 -- Daniel Baumann <daniel@debian.org>  Wed, 02 Jul 2025 05:38:28 +0200