Reading time: 4 minutes
Poor crawling is limiting your SEO potential. What are you doing about it?
What’s the point in having pages live on your website when search engines aren’t crawling those pages? Especially if your content is time-sensitive.
By the end of this issue, you will be able to:
- Troubleshoot crawling errors
- Learn how to get all pages crawled
- Understand the reasons behind traffic drops
You tried several things to outrank your competition, but everything was in vain. It’s frustrating to see the competition outrank you for the most important keywords.
A web crawler, as the name suggests, crawls your pages & saves them in the search index. This information is served to the user based on their search query. The rank depends on several ranking factors which is a discussion for another day.
The whole process of SEO can be summarized in 3 fundamental steps – Crawling, Indexing, and Ranking.
You can’t take the most fundamental steps of SEO, lightly. If your content doesn’t get crawled, there’s no way you can expect those pages to get indexed & rank on SERPs of any search engine.
It’s important to optimize your pages and make it easy for search engines to crawl your pages.
Search engines like Google crawl everything that can be indexed on the SERPs. If you manage the SEO for your site, you should know the file types that are indexable on Google.
If you’re aware of what is causing the traffic drops, you will be able to fix it before it creates irreparable damage.
Here are three things you can do to ensure there are little to no crawling issues for your site.
Table of Contents
Step 1: Learn how to get all your pages crawled
The first few hours are very important for any page. No matter how big the site is. Indexing & ranking takes time so crawling becomes the most important step for an SEO campaign.
Even for a published page, when you make any changes, it’s important to get your pages to crawl so that the updated version of the page loads for the end users.
Google loves fresh content and if there are any issues with crawling, your pages are never ranking again.
I recently wrote a post on LinkedIn on how I get almost all my pages crawled. I have covered steps that you can do right now so check it out.
Step 2: Troubleshoot individual pages
Google Search Console is the best tool to manage the crawling of your site. GSC reports any crawling errors right in the dashboard so that you can make the necessary changes. This is what being eligible for the SERP battle looks like.
Efficient crawling means reducing the time between publishing/updating pages and seeing the pages ranked (irrespective of position) on SERPs.
A large portion of crawling depends on the sitemap (I talked about it here). If the report shows that certain pages aren’t indexed, look for the reasons that the report shows. Here’s what indexing reports look like.
In most cases, you will be focusing only on “Crawled – currently not indexed” & “Discovered – currently not indexed”. Click on the reasons to see the list of pages that are affected by the reason.
Focus on “crawled – currently not indexed” URLs. Google will list pages that its crawler has crawled but not indexed because you did not explicitly request indexing.
3: Why traffic drops? What can you do?
Traffic drops are inevitable. The reasons can vary from algorithm to technical issues. Check the protocol (https or http) you have added to GSC and the actual protocol that is live.
The most common reason that non-technical online business owners have is tracking the wrong protocol on GSC. Make sure your live site and what’s added on GSC is the same.
You can also check if the pages you have updated are requested indexing again. Keep a close look at the Crawl stats report to ensure none of your pages are left uncrawled.
Check if you have accidentally added a no-index tag, especially if you use WordPress plugins to manage interlinks or indexing.
If you have moved a page to a new URL, make sure you have added 301 redirections. Even seasonal drops can be a reason why you have traffic drops.
Out of all these reasons, there are very few cases where crawling is the reason. Crawl stats reports is your best friend to deal with crawling issues.
Video Guide
The crawl budget is one of the most misunderstood concepts in SEO. This episode of SEO mythbusting discusses a lot about crawl budget.
Today’s action step → Check the crawl report & find pages that are crawled but not indexed. Request indexing for those pages.
Creator Spotlight
Sarvesh Shrivastava has generated 300k visitors a month in less than a year. Someone of that caliber has shared an expert opinion with you. I asked him what’s the single most important thing that one should know about crawling, Here’s what he has to say:
Tips for founders/solopreneurs
If you’re a founder & SEO is your priority, here are some content discovery secrets Sarvesh Shrivastava has shared:
- Only when you have a decent amount of content on the site, start thinking about optimizing for crawling.
- Submit your sitemap to Google Search Console.
- Have a mobile-friendly responsive design.
- Be patient. It can take some time for search engines to crawl and index your website.
SEO this week
- More users around the globe will see SGE on their smartphones. Google officially announces
- Martin Splitt shares timeless wisdom on how to optimize your pages for Core Web Vitals
- 302 or 301? Which type of redirect should you use? John Mueller answers
- After making FAQs for “authoritative site” only, Google is testing Mentioned in dropdown in SERPs
- While AI content testing tools are on the rise, Google released a tool to detect AI-generated images
Masters of SEO
- Customers don’t Google search with a credit card in hand. Andrew Holland shares how customers buy
- How to make $30,000 by spending $1500? Nick Jordan shares his personal experience
- How to dominate categories on SERPs? Saurav Agarwal teaches in this post
- How to save money if you work with content writers? Gordana Sretenovic wasted $50k so that you don’t have to
- Legit ways to make money in the SEO industry right now?
How can I help you?
I put a lot of effort into coming up with a single edition of this newsletter. I want to help you in every possible way. But I can do only so much by myself. I want you to tell me what you need help with. You can get in touch with me on LinkedIn, Twitter & Email to share your thoughts & questions that you want to be addressed. I’d be more than happy to help.
Whenever you’re ready to dominate SERPs, here’s how I can help:
- Sit with you 1-on-1 & create a content marketing strategy for your startup. Hire me for consulting
- Write blogs, social posts, and emails for you. Get in touch here with queries (Please mention you found this email in the newsletter to get noticed quickly)
- Join my tribe on Twitter & LinkedIn where I share SEO tips (every single day) & teaser of the next issue of Letters ByDavey