How to submit a crawl request.
What is a crawl request
Before we talk about how to submit a crawl request let’s quickly go over what a crawl request is. A crawl request is simply asking Google to re-index, or index for the first time, a page on your website. Google will do this periodically on its own but can take up to 4 weeks to complete. If you have time sensitive content to share with the world this Is unacceptable! To help expedite the crawling process you’ll want to submit a crawl request.
You’ve just made some changes to some pages on your website and you want to make sure google notices these changes as soon as possible. Thankfully, if you have Google Search Console set up, submitting a crawl request a pretty straightforward task. There are two methods to submit a crawl request.
URL Inspection Tool
The URL Inspection tool allows you to submit individual URLs to Google. This can be pretty time consuming if you have a lot of URLs. If that’s you then skip this section and dive right into sitemaps.
The URL inspection tool is located in the left side navigation panel in Search Console.
Once selected your browser will turn grey highlighting the search box at the top of your screen. Here you will enter the URL you wish to add submit to Google for indexing.
Make sure you type your URL exactly as it appears including HTTP or HTTPS. After entering your URL you will be show the URL Inspection results page. This will let you know if the URL is on Google already, if the page has been indexed, and whether or not your page is mobile friendly. If the page is not on Google or has changed recently you can select the REQUEST INDEXING button.
After requesting you’ll get a confirmation box letting you know that your request has been submitted. It’s important to note that (the dialog box mentions this as well) that submitting your request multiple times will not speed the crawl time.
A sitemap is my preferred method for submitting multiple pages (or entire websites) to Google for indexing. There are a three ways to submit a sitemap to Google. You can use the sitemaps report, your robots.txt file, or the ping tool.
The first step is obviously to create a sitemap. If you are unsure or unfamiliar with them, you can check out my previous post on how to make a sitemap. I also cover how to submit the sitemap via the sitemaps report, so I won’t cover that here.
The second method is to add a short line of text to your robots.txt file. This lets crawlers know where to find your sitemap before they start to crawl your website. All you need to do is add the following line to your robots.txt file:
You might be thinking, what the French toast is a robots.txt file and how do I make one? Well don’t worry I’ve got you covered. Check out another one of my previous posts about robots.txt. I cover what it is, what it does, and provide some useful tips for SEO best practices.
The final method to submit your sitemap is with the ping tool. The ping tool allows you simply to place a line of text in your browser or command line to let google know where your sitemap is located. All you must do is type the following:
That’s it! That’s all there is to submitting a crawl request to google. Remember if you have one or only a few URLs then you can easily use the URL inspection tool in Google Search Console. If you have a large number or want to have your whole site recrawled, then you can submit a sitemap via the sitemaps report, the ping tool, or adding a line of code to your robots.txt
SEO FOR WEB DESIGNERS COURSE
Get on the waitlist to learn more about SEO for Web Designers and be notified when it's open for enrollment!