A lot of different skills go into being a great SEO. On the one hand, you need to have classic marketing skills that allow you to empathize with a prospective customer, speak to their needs and wants and understand their behavior. On the other hand, you must have at least a working knowledge of web technology, including HTML and server administration, that will allow you to properly publish content on the Internet.
This latter is referred to with the blanket term “Technical SEO”. In this article I’m going to give a brief introduction to technical analysis for SEO purposes, including an intro to my favorite SEO tool on the market, the Screaming Frog SEO Spider.
Technical SEO Analysis
Before we dive into the tactics of how to use a crawler, let’s go over some of the basics of SEO. When analyzing on-site and on-page SEO, we are concerned with the following general areas:
- On-page Content (Text, Images/Alt Tags).
- URL Structure (Keyword inclusion in URLs, not too long, etc).
- Meta Data (Title Tags, Meta Descriptions, Canonical Tags, etc).
- Server Errors or Redirects (i.e. 404 errors, 301 redirects).
- Internal Linking (how the website links to itself).
Let’s break down quickly what we want from an ideal website when it comes to these areas:
On-Page Content
Speaking very generally here, you will want lots of text, images and other media. The exact structure and content of your page will depend on your niche, but will always content text and images. (PRO TIP: To see what Google is favoring, just check out what the top 5 ranking sites are doing, and mimic that).
URL Structure
No need to go crazy here: Use keywords in your URLs, and don’t make them crazy long, you’ll be fine.
Meta Data
Title Tags are the most important thing to keep in mind here, they should be well crafted to include 1 main keyword, plus a few other secondary keywords, along with your brand name. “Well crafted” means making a sentence that makes sense to humans (NOT “red widget Blue widgets green widgets widgets for sale”)
Meta data also includes your meta description (sales copy, helps increase click-thru-rates from SERPs), and other data associated with each URL, like canonical URL, robots tags, etc.
Server Errors and Redirects
Server errors happen when a URL is pointing to a resource that doesn’t exist. Redirects are used purposefully when old resources move to a new location, BUT you should never rely on redirects for your internal linking (doesn’t make sense, just update the links to point to correct URLs)
This is the most important thing I look for when doing an initial review, usually you can find a few items you can fix quickly, such as a malformed links that lead to 404 pages, or unintentional 301 redirects.
Internal Linking
Ah, links. The focus of so much SEO effort. Internal linking, just like external links (i.e. backlinks) is extremely important, but you don’t have to go crazy trying to figure out the best approach. As a simple rule: Make sure all your pages can be reached by a simple web crawl, starting at your homepage.
Ok, now that we’ve got the basics out of the way, let’s dive into using Screaming Frog to check out all these elements out…
Crawling and Technical SEO Analysis with Screaming Frog
So go on and download Screaming Frog, don’t worry, it’s free (although you can only crawl 500 URLs at a time). I’ve started it up and plugged in “https://www.ebay.com”
Neat, eh?
Let’s break down what it’s done here:
- It fetched https://www.ebay.com.
- Captures all the appropriate data for that URL.
- Captures all the URLs that are linked on https://www.ebay.com, adds those to the queue to get crawl next.
- Rinse and repeat.
Hypothetically, if you left the crawl running, you would capture every page on Ebay.com (unless of course, they are “orphaned”, i.e. not linked from anywhere, which is something we will check for in our analysis.)
So how would we go about addresses our technical SEO areas with screaming frog? Easy, let’s go into it:
On-page Content
You can get a very high-level view of on-page content. It won’t diagnose quality issues or duplicate content issues, but it will let you see word count, H1/H2/etc. tags. These can be easily scanned for quality issues.
It also has something that’s so useful I must mention it every time I speak about this stuff. You can download a list of all the Images that are missing alt text. Just go to: Export -> Images -> Images without alt text inlinks.
URL Structure
This one is easy, you can see all your URLs in a handy spreadsheet. This is great for spotting any weird parameters, any broken URLs, checking for keywords and URL length.
Meta Data
Meta data can be easily checked by either exporting the data to a spreadsheet, or scrolling through the tool. The key data points you want to check are:
- Title Tags
- Meta Descriptions
- Canonical Tags
- Meta Robots
- Server Errors
The first thing I check when running these audits is for inadvertent 404 and 301 response codes. Every internal link should ideally be pointing to a 200 URL. If there are any non-200 response codes, be sure to investigate the cause. Often it will be because of a broken link that is easily fixed.
Internal Linking
Internal linking is the cause of much debate in SEO circles. There’s a million different “Silo” techniques, but I’ve come to ignore specific silo’ing methods and instead just focus on making sure your pages are heavily interlinked.
In a quick audit I will usually do the following:
- Let the crawl run to its completion.
- See how many HTML pages are in the final crawl.
Go to google.com and type the site: command for you domain (i.e. site:example.com). Take note of the results.
Compare those numbers. Are they way off? It means you either have orphaned pages, or google isn’t including some pages of yours for some reason.
For a deeper analysis you can check each of your money pages for their “inlinks” (i.e what internal links are pointing to it), and see where there might be room for improvement. That data is readily available by highlighting a URL and using the lower pain of the screaming frog window to browse through that URLs link data.
Conclusion
Technical analysis is all about viewing a website how a search engine spider (i.e. Googlebot) would view it. This can be done manually, by checking source code, clicking links, etc., but it is tedious, prone to human error and never comprehensive on larger sites.
Therefore, it is imperative that an analyst use an automated tool (such as Screaming Frog), to crawl and collect data on a website. It will give a bird’s eye view of the status of the website and allow for relatively easy diagnosis of any issues.
Happy crawling!
About the author:
Michael Hayes founded Darby Hayes Consulting in 2015 to help business owners grow their revenue through aggressive digital marketing. He expertise is in SEO, having worked in the industry since 2008. He can be reached at mike (at) darbyhayesconsulting.com.