![\"whats](\"https:\/\/storage.googleapis.com\/dopingcloud\/blog\/en\/2021\/06\/whats-the-deal-wilth-sitemaps.png\")
<\/figure><\/div>\n\n\nBefore diving into the robots.txt sitemap topic, we need to remember a few elements about robots.txt files. You may picture robots.txt files as a kind of intelligent guide or instruction manual for search engine robots. They give indications about which pages should or should not be crawled and also determine which robots are allowed to crawl your website. <\/p>\n\n\n\n
In other words, robots.txt is a text file or core component of a website. It can come in different formats, depending on the goal it is supposed to serve. For example, if you don’t want your site to be too restrictive or selective, the ‘User-agent’ and ‘Disallow’ areas of the file can be left almost blank.<\/p>\n\n\n\n
On the other hand, if you intend to give specific instructions to crawling robots, then you will have to fill those areas with further information about which ‘user-agents’ you want to provide access to or which URLs you ‘disallow.’ One thing to keep in mind though: you can’t control the bots 100%. Even if they follow the indications written in your files, they will also decide what to do on their own if your files are confusing or lacking some mandatory details. <\/p>\n\n\n\n
Speaking of more information, let’s not forget that each of your subdomains requires separate robot.txt files. So let’s say that your domain name is inflatablemugs.com: you will need a file for the domain, but also for the subdomains related to it (e.g., contact.inflatablemugs.com, about.inflatablemugs.com, and so on).<\/p>\n\n\n
<\/figure><\/div>\n\n\n