256 Kilobytes

How to crawl a sitemap with Screaming Frog

Articles in Search Engine Optimization | By Hash Brown

Published 2 months agoMon, 04 Feb 2019 20:41:06 -0800 | Last update 2 months agoMon, 04 Feb 2019 20:41:54 -0800

Screaming Frog is a great tool for SEO, helping SEO's fix errors and run audits on their website. Today we will be looking at a feature of Screaming Frog.

491 views, 0 RAMs, and 0 comments

Screaming Frog is a great tool for all sorts of uses.

It's basically a tamed spider that crawls your website exactly as Google Bot would, except it gives you the data from the crawl in the form of a report allowing you to fix errors.

There are times when letting the Screaming Frog spider crawl your website on it's own won't really do the job, examples of this include:

  • You're crawling someone elses website
  • You only want to crawl a part of your website
  • You're scraping content and only want to visit the URL's where this content is contained.
  • You're website has orphan pages or subdomains
  • You're testing a website before launch so linking/archive pages are not available
  • Fuck it, some other retarded reason

Screaming Frog DDOS

To do this you may find yourself wishing you can crawl the website via the sitemap, well you can!

Step 1: Find the sitemap

I'm going to assume everyone is a responsible person and they are crawling their own website, so finding sitemaps should be very easy. However for those who are not responsible or you don't know where your sitemap is, viewing the robot.txt file of most websites will lead you directly to this.

This can be found at:
domain.com/robots.txt

You can also use google, a search similar to this will probably find most sitemaps.

site:domain.com sitemap xml

Step 2: Tell Screaming Frog where the sitemap is

Now we know where this is, lets load it into Screaming Frog.

Head to "Configuration" > "Spider", you will see this screen.

Screaming Frog Sitemap Crawling

Note: If you can't see this, please update your program.

Load your sitemaps into the box highlighted, if you have multiple sitemaps just put one per line.

Click "OK" when complete.

Step 3: Crawl baby crawl

Now Screaming Frog knows what to do, enter the domain in the crawl field and click "Start".

Screaming Frog will now be crawling your sitemap(s) loaded in the previous step.

Download more RAM. 🐏 ⨉ 0 Posted by Hash Brown 2 months ago

Edit History

• [2019-02-04 20:41 PST] Hash Brown (2 months ago)
🕓 Posted at 04 February, 2019 20:41 PM PST

Profile Photo - Hash Brown Hash Brown
🗎 41 🗨 270 🐏 89
Staff

I'm Hash Brown, I've done "computer stuff" for all my life.

Here are some of my latest articles:

Marketing:

Development:

Programming:

Misc:


Account created 4 months ago.
41 posts, 270 comments, and 89 RAMs.

Last active 3 days ago:
Commented in thread Best Subreddits about SEO?

Post a New Comment

To leave a comment, login to your account or create an account.

Do you like having a good time?

Read Quality Articles

Read some quality articles. If you can manage to not get banned for like five minutes, you can even post your own articles.

View Articles →

Argue with People on the Internet

Use your account to explain why people are wrong on the Internet forum.

View Forum →

Vandalize the Wiki

Or don't. I'm not your dad.

View Wiki →

Ask and/or Answer Questions

If someone asks a terrible question, post a LMGTFY link.

View Answers →

Make Some Money

Hire freelancers and/or advertise your goods and/or services. Hire people directly. We're not a middleman or your dad. Manage your own business transactions.

Register an Account
You can also login to an existing account or recover your password. All use of this site is subject to terms outlined in the terms of service and privacy policy.