Since the internet’s first appearance as the starting point of the Information Age in 1983, digital marketing has been growing, evolving, and adjusting to trends. In addition to the internet’s development, search engines are continually updating to provide the best results for a search query. So, with so much to keep an eye out to stand out in the massive numbers of websites, search engine optimization (SEO) has become the most prominent tool. Screaming Frog is one of the major players in SEO.
SEO specialists and the analysis they provide allow website owners to take action to adjust their websites to be visible on the search engine results page (SERP). However, as it sometimes becomes a difficult task to check every corner of a website individually, there are some great tools to help you with scanning and analyzing your website. In this article, we will go into the details of one of those tools: Screaming Frog SEO Spider Tool.
A dependable tool for advanced and technical
SEO, Screaming Frog SEO Spider Tool is an easy-to-use helper, and we will
explain how to use it in this article. Before we start learning how to use
Screaming Frog for SEO, we first need to learn what it is and what it does.
What Is Screaming Frog?
Screaming Frog SEO Spider Tool is one of the most reliable and advanced SEO tools which provides detailed and helpful information on adjusting and improving a website. Just as search engine spiders crawl every web page existing on the internet, Screaming Frog SEO Spider Tool crawls a website’s images, backlinks, CSS, content, and many others as a desktop application. It basically crawls a website just as a search engine crawler would and shows you what the crawler would see, searching for and filtering out existing SEO issues. Thus, it provides SEO specialists and website owners with bad redirects, duplicate pages or content, missing metadata, and a lot more helpful insights to improve their ranking on SERP.
Here is a list of what the SEO tool crawls on
- Internal and external links
- Meta refresh
- Links outside of the start folder
- All subdomains
- Internal or external “nofollow”
- XML sitemaps
Screaming Frog SEO Spider Tool offers everything you can ask for while doing a site audit (including the backlink, XML sitemaps, pagination (rel= “next” and rel= “prev”), hreflang, and canonicals audits which we will explain how further in the article). The results can help you develop your site, and SEO works to perform more qualified in searches.
Now that we have tried to explain what
Screaming Frog SEO Spider Tool is and what it does in a concise manner, it is
finally time to learn how to do a Screaming Frog analysis for SEO.
How to Do Screaming Frog SEO Analysis?
After installing the desktop application from
Screaming Frog’s original website, we recommend getting familiar with the
application’s menu and items before touching anything. Also, we highly
recommend using the full version as it is more than helpful in configuring your
The menu tabs are:
- File: The tab where you can save a crawl as a file to have the data for further analysis. The tool also saves the last six crawls in its database just in case you forget to save.
- Bulk Export: The option to export the crawled data quickly. It also allows you to export addresses with specific response codes like 401s, directives, anchor text, images, inlinks, and more.
- Reports: It is the menu tab that allows downloading of the crawling as a report. Also, you will be provided with data such as redirect chains or canonical errors.
- Sitemaps: This menu tab allows you to construct a sitemap.
Now that you have learned how to operate the
desktop application, it is time to learn the crawling process.
SEO Analysis Using Screaming Frog SEO Spider Tool
Now that you are ready, paste the URL of the
website you want to be crawled and hit the Start button. However, the progress
may take a couple of minutes, depending on the size of the website.
1-Checking the Response Code
Clicking the “Response Codes” tab
will provide all response codes to the crawled website’s URLs. The response
codes you should keep an eye out are:
- 200 OK, indicating that the request has been successful.
- 301 Moved Permanently
- 302 Found, previously called “Moved Temporarily”
- 404 Not Found, indicating that the content can not be viewed at present but can be found in the future.
- 500 Internal Server Error appears when an unexpected condition is met.
- 503 Service Unavailable indicates that the server can not handle the request. It could be because the server is overloaded, or it is down for maintenance.
Briefly, the 2xx, 3xx, 4xx, and 5xx codes are
success, redirection, client error, and server error codes in order. First, you
should recognize all the pages with 404 error codes, and 301 redirect them to a
related page. You can also check for broken internal and external links on the
Response Codes tab. After selecting a URL with a 4xx or 5xx code, click the
“Inlinks” or “Outlinks” tab to view the broken links.
Moreover, updating all the web pages with
redirects and error codes will impact the user experience and significantly
improve your rankings.
2-Checking the URL Structures
The importance of a good URL structure and its
influence on SEO is a crystal-clear fact. Screaming Frog SEO Spider Tool will
analyze your website’s URL structure under the “URL” tab. It is
significant to analyze the URL structure to see URLs that contain unusual
parameters and characters that are illegible for search engine crawlers to
crawl and rank.
In addition, making sure that the URL contains
four to five words, stands descriptive and unique will provide a better user
3-Correcting Page Titles
Moving on to the “Page Title” tab,
you will see all the page titles on your website with their pixel length and
word count. Besides, Screaming Frog SEP Spider Tool lets you filter the titles
that remain as a problem. For example:
- Duplicate Titles: If you see two titles that are the same, correcting it will improve your SEO.
- Keyword Optimization: Ensuring your web pages contain the most crucial keywords in your field, along with avoiding the over-excessive usage of keywords, is also essential for your SEO.
- Title Length: Once you use the “Character Column,” you will be able to separate titles that are longer than 50-65 characters. Longer titles will impact the visibility on SERP, while shorter ones will not be appealing to your targeted audience.
4-Analyzing Meta Descriptions
Meta descriptions appear on the search engine
result pages under the page title. Ensuring that it does not exceed the
160-characters limit is crucial. Also, checking that the meta description does
not contain duplicate content and optimized by keywords is significant to get
5-Testing SERP Snippets
One of the most important elements that appear
on SERP, which impacts the ranking is, of course, a snippet. The SEO Spider
Tool also provides how your snippet would appear on the SERP. This way, you can
continue editing your URL, page title, and meta description until it fits your
Unfortunately, images on your web pages do more
than improving visual quality. However, it has a significant part in SEO and
should be optimized carefully. Since page speed is essential both for mobile
users and search engines, it is best to keep the images under 100kb. So, larger
images will hurt page speed, and it will eventually result in lowered rankings.
Via Screaming Frog, you will be able to see the
images that are over 100kb, without alt texts, and containing alt texts over
Thanks to the advanced SEO tool, Screaming Frog
has a “Directives” tab that allows you to check technical issues. You
can filter the web addresses that contain canonical, no canonical, follow,
nofollow, index, noindex, and other significant elements. Besides, you will be
able to see how robots.txt is working on your website and make adjustments.
8-Analyzing Crawl Depth
The crawl depth of a URL is the number of
clicks that takes a user to get to a specific page from the homepage. However,
the crawl depth will change according to a website’s size.
If the crawl depth on the SEO tool is high, it
will be harder for your website to get ranked. For example, let’s say that
Screaming Frog indicates that an essential product on your website has a crawl
depth of 5 after crawling your website. It will be better to change the
website’s organization and hierarchy to get the crawl depth to 2.
9-Generating XML Sitemap
An XML sitemap is an important element to get
search engines to understand your website better. In other words, it is a
roadmap for crawlers to crawl your website and ensure that everything is
You will find the “Create XML
Sitemap” feature in the top navigation bar under “Sitemaps.” You
can manually adjust the preference and modify the rotation of individual pages.
Types of Audits in Screaming Frog SEO Spider Tool
An audit is a sub-set of SEO analysis because
it means you are checking for apparent technical issues. These issues are known
as red signals for the indexing and ranking of your website by Google. For this
reason, Screaming Frog supports doing audits in its application. Here you can
find brief guides on how to do each audit.
The SEO Spider will crawl canonical link
elements found in the HTML and HTTP Headers and report in their standard
- Enable ‘Store’ and ‘Crawl’ canonicals via “Configuration>Spider>Crawl.”
- Enter the website URL and start the crawling.
- View the “Canonicals” tab. Filters ‘contains canonical’, ‘self-referencing’, ‘canonicalized’, ‘missing’, ‘multiple’, and ‘non-indexable canonical’ are available.
- View’ Indexability Status’ of non-indexable canonical URLs via the “URL Info” tab.
- Bulk export source URLs, non-indexable canonical URLs, and response codes via “Reports>Canonicals>Non-Indexable Canonicals.”
- View chained canonicals and loops via “Reports>Canonicals>Canonical Chains.”
The SEO Spider will crawl rel=”
alternate” hreflang explanations in HTML, via HTTP Header or in XML
- Enable ‘Crawl’ and ‘Store’ hreflang via “Config>Spider>Crawl.”
- Select ‘Crawl Linked XML Sitemaps’ to crawl hreflang in XML sitemaps via “Config>Spider>Crawl.”
- Crawl the website URL.
- Populate hreflang filters on the “Hreflang” tab via “Crawl Analysis>Start.”
- View errors via the “URL Info” tab.
- Bulk export source URLs and errors via “Reports>Hreflang.”
Rel=”next” and Rel=”prev” Audit
The SEO Spider will crawl pagination
- Via “Configuration>Spider,” select ‘Crawl’ and ‘Store’ pagination.
- Start crawling.
- Populate pagination filters on the “Pagination” tab via “Crawl Analysis>Start.”
- Bulk export via “Reports>Pagination.”
XML Sitemaps Audit
- Enable ‘Crawl Linked XML Sitemaps’ via “Config>Spider>Crawl.”
- Start crawling.
- Populate sitemap filters on the “Sitemaps” tab via “Crawl Analysis>Start.”
- Click “Inlinks” to view the XML sitemap source.
- Bulk export the report.
- Config XPath custom extraction via “Config>Custom>Extraction.”
- Then, switch to list mode via “Mode>List.”
- View the URLs that are blocked by robots.txt via “Config>Spider>Basic.”
- Upload the backlinks and start the crawling.
- You will be able to review the report and then bulk export.
To Sum Up
With the Screaming Frog SEO Spider Tool, you can analyze the most critical elements. It is an exceptional tool to help you optimize a website and boost its performance in search result pages. Additionally; it should be a necessary tool every web designer and SEO specialist should operate.