robots.txt

robots.txt is a file which is usually placed in the root of your website. It decides whether crawlers are permitted or forbidden access to your website.

For example, you can forbid crawlers to visit a certain folder (and all the files therein contained) or to crawl a specific file, usually to prevent those files being indexed by other search engines.

Settings#

No settings.

Generated file#

Thulite SEO generates the following robots.txt — for example:

User-agent: *
Disallow: /admin/
Sitemap: https://seo.thulite.io/sitemap.xml

Resources#

• How Google interprets the robots.txt specification