Page indexing and link indexing are important topics in SEO and are often overlooked and misunderstood by beginners. Indexing can be a critical step in link building and is also important for your website.
Before reading this post, we would suggest to read our blog what are backlinks to have some better context.
WHAT IS LINK INDEXING | INDEXING AND CRAWLING
For Google, the whole process starts by crawling: bots, called googlebots, crawling the web for pages. Google has different bots for mobile, desktop, video, photo, news.
After crawling the pages are rendered and finally, if content is worthy for Google, added to the index. The process is not as straightforward as it seems….
New content is indexed faster when linked to published content. That is why orphaned pages are difficult to index.
Google has two separate indexes (primary and secondary) which are the desktop index and the mobile index.
in 2016, Google announced that the primary index was switched from desktop to mobile. This was done to reflect the trend that most searches were now done with mobile phones. You can check this by yourself by looking at your website’s Google analytics.
A consequence of this change is that mobile search results are slightly different than desktop search results.
This also means that mobile search results reflect more quickly changes being made to your website.
Indexing is the process by which google extracts the most relevant information and adds them to the index. When you execute a search, Google displays the best information based on the index. The index is similar to a library where billions of web pages are stored and organized. Actually, Google’s index size is not as big as we might think….. As many pages on the internet are spam or do not carry useful information the goal for Google is have the best index possible and not the largest.
It was confirmed recently that Google does not crawl the whole content of a website and therefore index a limited number of pages (https://www.searchenginejournal.com/google-not-indexing-site/416717/)
These information about indexing are now displayed on your google search console as pages discovered but not indexed. Content can be crawled and indexed later, or not….
GOOGLE SEARCH CONSOLE | LOOKING FOR INDEXING ISSUES
Google search console is a powerful tool to check if you currently have indexing issues with your website. On the left handside you will found the Index–> coverage section.
Pages that Google did not include in the index, are presented in different categories:
Redirect error
Excluded by ‘noindex’ tag
Crawled – currently not indexed
Page with redirect
Alternate page with proper canonical tag
Not found (404)
Discovered – currently not indexed
Soft 404
Duplicate, Google chose different canonical than user
Duplicate, submitted URL not selected as canonical
CAN MY BACKLINKS BE INDEXED?
The first thing to check is if the robots.txt file allows bots to visit the page. If robots are blocked from crawling pages with the disallow command, the message below will be displayed by Google mobile-friendly test. If a page is blocked, it can neither be crawled nor fetched by robots.
Once you have made sure that a page can be crawled, you have to check if this page does not have a noindex tag.
The noindex tag will prevent a page to appear on Google search results.
As you can see on the screenshot below, a page with the no-index can still be crawled and fetched by Googlebot smartphone.
HOW TO INDEX BACKLINKS | WHAT ARE INDEXERS?
Link indexing is a task that is getting more and more difficult in 2021 and 2022 for SEO agencies and freelancers. Numerous threads of complaints of low success rate can be found in SEO forums. What took days to index before can take weeks now. We experienced our first difficulties in indexing links back in February 2020 and it did not get better in 2021 and 2022, far from it….
To increase indexing rates, some people have developed indexers. Indexers are platforms built by a third party to force the indexing process. Few of them are free, and as expected, have a low success rate. In 2022, you can find more than 10 paid indexing services on the marketplace which have different indexing rates.
Indexers usually propose monthly packages to index thousand of links but can work also with a more flexible credit system if you have to index a small amount of links over few months. Sending a link to an indexer costs between 0.01 and 0.02 USD.
WHAT IS LINK INDEXING | INDEXING AND DEINDEXING
It is not because a link has been indexed that it will stay indexed forever. We noticed on few occasions that links were being deindexed by Google.
Few reasons can come to mind to explain this: either links were not well connected or Google finally considered that these links were not relevant and not worth being in the index.
It is also possible that the hosting platform where the backlinks are decided to add noindex tags. It was the case for the publishing platform Medium which started implementing the noindex tag in the first half of 2021.
DO BACKLINKS NEED TO BE INDEXED TO PASS POWER?
Getting webpages crawled and indexed is very important so they can appear on search results and start getting organic traffic and conversions.
For parasite SEO, indexing is a key step as the goal is to have your parasite showing up on the SERP.
For backlinks, is indexing as important as webpages and parasites?
As we discussed before, backlinks are important in SEO. One much debated topic in SEO forums and groups is if do-follow backlinks need to be crawled or fully indexed to count for SEO e.g pass power.
Many think that backlinks need to be fully indexed to pass power to the money website. A minority seems to think that getting backlinks crawled by bots is enough.
Finally some think that links crawled by bots counts for SEO but as not much as the ones fully indexed.
HOW DO I CHECK IF MY PAGES OR LINKS ARE BEING INDEXED?
If you do not want to wait for your links to appear on Google Search Console, Semrush or Ahrefs dashboards, you can check links one by one by performing one of the three searches below on Google:
http://www.link1.com (leads to broader results)
inurl:http://www.link1.com
site:http://www.link1.com
This technique is not always precise as sometime a link do not show up on google search but does on Ahref or Semrush.
When you want to check the backlinks of your money site or another page, you will notice some differences between Semrush and Ahrefs so it is good to use both.
If you have a list to check, zenserp has a free Google Index Checker Tool and gives some decent results.
Another index checker by Linkody is Index Checkr which has a paid version to check links in bulk for as low as $0.0011 per check.
HOW FAST MY NEW CONTENT OR LINKS CAN BE INDEXED?
Web pages can take few hours, days or even weeks to get indexed. Your content will be approved very fast if your website is Google News approved, . To speed up the indexing process you can request bots to visit your pages via Google Search Console but results are not guaranteed.
As mentioned earlier, indexing backlinks is more difficult than indexing your web pages and SEO indexers are widely used for this purpose .We noticed that indexing issues on Google were linked to Google updates. SemRush has a useful tool called Semrush sensor which measures the SERP’s volatility and therefore an indication of Google updates (even non announced). If volatility is high, it is likely that indexing remains difficult so it is better to wait until the volatility index comes down to send links to your favorite indexer.