Robots.txt improvements.

- Replace file with configuration snippet.
- Allow some domains, disallow others.