{"id":5686,"date":"2021-06-10T08:31:20","date_gmt":"2021-06-10T08:31:20","guid":{"rendered":"https:\/\/www.dopinger.com\/blog\/?p=5686"},"modified":"2024-08-12T12:46:05","modified_gmt":"2024-08-12T12:46:05","slug":"meta-robots-tag-guide","status":"publish","type":"post","link":"https:\/\/www.dopinger.com\/blog\/meta-robots-tag-guide","title":{"rendered":"Meta Robots Tag Guide"},"content":{"rendered":"\n

The meta robots tag is said to be one of the essential SEO tags to audit<\/a> your pages! After all, your page is tagged with the robot’s meta tag “noindex,” for example, no matter how valuable, complete and practical your content is, it won’t show up on Google.<\/p>\n\n\n\n

The name seems difficult, but it believes me. This goal is much simpler to validate and understand than it sounds.
Within this context, we intend to tell you a little more about the robot’s meta tag, how to use it, and the most common commands you can use to have greater control over the indexing of your pages on Google! In the end, we will give you some tips for not misusing the meta tag and in which cases you can apply each command.<\/p>\n\n\n\n

<\/span>What Is Meta Robots Tag?<\/span><\/h2>\n\n\n\n

Robots directives are pieces of code \u2013 simple \u2013 added to your website to direct robots like Google on crawling and indexing your website’s contents. The robot’s meta is part of this directive and is HTML code added to the head of the pages, as shown above! Besides it, there is another one called Robots.txt, which is a file added to the site’s root to communicate with search engine robots.<\/p>\n\n\n\n

The difference between the two is that the first \u2013 the subject of this post \u2013 serves to control your content’s index on Google and other search engines. The second controls the tracking, that is, which website content you allow robots to access. Both are very confused. After all, they have a similar name and similar purposes. But make no mistake, its features are different.<\/p>\n\n\n\n

The robots.txt file can prevent a URL from being indexed simply because Google can’t access its content, but that doesn’t mean it’s blocked from appearing in the results. If you have a link on the site to it, for example, it can be indexed “by accident.” The purpose of the file is to keep Google from wasting time on useless pages on your site, to avoid crawling issues, and let Google focus on what matters! The robot’s meta tag directly controls how and whether your page appears in search results.<\/p>\n\n\n

\n
\"how<\/figure><\/div>\n\n\n

<\/span>“How” vs “If”<\/span><\/h2>\n\n\n\n

What Do You Mean by “How” and “If”? Let’s find out.<\/p>\n\n\n\n

How?<\/h3>\n\n\n\n

Through meta robots, you can, for example, control the size that Google should display your images in places like Google Discover. You can also block your pages from appearing in featured snippets and various other possibilities.<\/p>\n\n\n\n

If<\/h3>\n\n\n\n

With the directive index or noindex \u2013 let me explain each one better \u2013 you can control whether Google can show your page on SERP<\/a> or not!<\/p>\n\n\n\n

<\/span>Explaining Meta Robots Names and Directives<\/span><\/h2>\n\n\n\n

Let’s break a standard meta robots code into parts so you can better understand how it works:
The “name” field, despite frequently appearing as “robots,” can be changed so that the target is directed to specific robots, in particular cases, such as:<\/p>\n\n\n\n

name=”Googlebot-Image”<\/p>\n\n\n\n

The “content” field is where the magic of the goal of the robot happens \u2026You need to be familiar with each of the policies that can be added to handle the goal. Find out now what are the main directives for controlling the Google robot: <\/p>\n\n\n\n

Note: it is noteworthy that these directives are not exclusive and can be worked together by separating commas, for example: “index , nofollow<\/a>” – in this case, we allowed the page to be indexed, but its links cannot be tracked for Google.<\/p>\n\n\n\n

All<\/h3>\n\n\n\n

It is the default value of the policy: index, follow.
In other words, by default, if you don’t have anything filled in the meta, the robots consider that the page can be indexed and have its links followed! No need to use this: meta name=”robots” content=”all”<\/p>\n\n\n\n

Noindex<\/h3>\n\n\n\n

The famous meta robots noindex . This is one of the most important directives, as it defines whether your page can be indexed on Google or not. In this case – noindex -, you block your page from appearing in the search results of Google and other search engines. meta name=”robots” content=”noindex”<\/p>\n\n\n

\n
\"nofollow\"<\/figure><\/div>\n\n\n

Nofollow<\/h3>\n\n\n\n

It prevents robots from tracking (accessing) links present on the page. So, for example, if you add this directive in this post, all the links you put in it won’t be followed by Google when you go here.
meta name=”robots” content=”nofollow”<\/p>\n\n\n\n

None<\/h3>\n\n\n\n

It combines noindex and nofollow (equivalent to adding: “noindex,nofollow\u201d ). That is, with this directive, Google cannot index or follow the links on the page. meta name=”robots” content=”none”<\/p>\n\n\n\n

Noarchive<\/h3>\n\n\n\n

This policy prevents Google from displaying a cached link in search results.<\/p>\n\n\n\n

What is Cache?<\/h3>\n\n\n\n

It’s an information repository, where, in Google’s case, pages are saved in Google’s repository so that users have quick access to it, even if it’s down.<\/p>\n\n\n\n

The cache policy can be implemented on your site independently of Google, so this policy can be useful on sites that prefer to display their versions of cached pages. meta name=”robots” content=”noarchive” It tells Google not to display a snippet of text or preview videos in search results.<\/p>\n\n\n\n

Nosnippet<\/h3>\n\n\n\n

For example, it prevents Google from displaying your pages’ meta descriptions in search results.
meta name=”robots” content=”nosnippet”<\/p>\n\n\n\n

Unavailable_After: [date\/time]<\/p>\n\n\n\n

Allows you to set a date\/time limit on when the page can be indexed on Google.
After the set date, the page can no longer appear in search results (useful for landing pages for temporary releases, for example):<\/p>\n\n\n\n

meta name=”robots” content=”unavailable_after: 2021-06-05″<\/p>\n\n\n\n

<\/span>How to Validate the Goal of the Robot<\/span><\/h2>\n\n\n\n

Here are the steps to validate the goal of the robot.<\/p>\n\n\n\n

    \n
  1. Navigate to Page: You can go to the page you want to validate and press CTRL + U, then CTRL + F and search for \u201c robots \u201d to see if and how the tag is appearing on the page.<\/li>\n\n\n\n
  2. Use Extensions: You can use a Chrome extension for SEO<\/a>, such as SEO META, in 1 CLICK.<\/li>\n\n\n\n
  3. Validate: To validate at scale, it is ideal to use a tracking tool such as Screaming Frog.<\/li>\n<\/ol>\n\n\n\n

    <\/span>How to Implement the Robots Meta Tag?<\/span><\/h2>\n\n\n\n

    This is a very simple HTML markup to add to the head of the pages, as in the example:
    <\/p>\n\n\n\n

    <head>\n   <title>T\u00edtulo da p\u00e1gina<\/title>\n   <meta name=\"description\" content=\"description page\" \/>\n   <meta name=\"robots\" content=\"follow, index, max-snippet:-1, max-video-preview:-1, max-image-preview:large\" \/>\n   <link rel=\"canonical\" href=\"url\" \/>\n<\/head><\/code><\/pre>\n\n\n\n
    To implement it, add it with the pattern above to all the pages you want for indexing in the way Google sees fit for users! But How do You do It? The answer is: it honestly depends. It depends on your CMS! Many already have it as default. Others don’t. It’s up to you to validate if it’s there and understand with a dev how it can add the markup to the pages’ HTML or yourself.<\/p>\n\n\n
    \n
    \"what<\/figure><\/div>\n\n\n
    <\/span>What is X-Robots-Tag?<\/span><\/h2>\n\n\n\n
    It’s not just the meta robots tag that controls indexing. There is another one, X-Robots-Tag, which has the same function and directives. The only difference is in the form of implementation. While the meta tag is a markup in the page’s HTML code, the X-Robots-Tag is an HTTP response \u2013 like the 301 redirect \u2013 sent by the website’s server.<\/p>\n\n\n\n
    It is useful in some specific cases, mainly to block the indexing of files like PDFs or images, for example.
    To validate the X-Robots-Tag, the process is slightly different, as you should see the HTTP response from the page.
    You can use the screaming frog to validate it to scale. For a quick check on the page, you need to use some Chrome extensions.<\/p>\n\n\n\n
    <\/span>7 Use Cases of Meta Robots Tag<\/span><\/h2>\n\n\n\n
    Learn how to use this tag with examples of the most common cases where it is useful:<\/p>\n\n\n\n