As announced on October 18th, 2021, Bing and Yandex are adopting IndexNow – a shared protocol that will allow website owners to notify these search engines whenever their website content is created, updated, or deleted. 
‍
In my opinion, it might be the most significant piece of news in the technical SEO world since Googlebot started using evergreen Chromium to render content, although it doesn’t change much for now.
‍
It’s a giant step towards limiting crawling activity in favor of indexing APIs. Whether or not Google follows suit, IndexNow is a sign of things to come and a clear signal that existing indexing pipelines are subject to change.

What is IndexNow

IndexNow is a protocol that allows site owners to notify multiple search engines of new or modified content on their websites.
‍
While we still don’t have all the details, it’s likely that Bing decided to build on its Content Submission API, released in September 2021, and create an open indexing protocol shared with other search engines that might want to participate.
‍
The official website states the following about the possibility of participation: “Search Engines adopting the IndexNow protocol agree that submitted URLs will be automatically shared with all other participating Search Engines. To participate, search engines must have a noticeable presence in at least one market.”

How to use IndexNow

To ping the participating search engines (at the moment, these include Bing and Yandex), you first need to verify the ownership of your domain.

There are two options to do so:

  1. Create a .txt file at the root directory of your host that contains a UTF-8 encoded key. The file also needs to have the key used as its name, like
    https://www.example.org/your-key.txt containing “your-key.”
  2. Create one or multiple text files in other locations within the same host and provide their location when submitting URLs for indexing.

Then, you can submit either a single URL or a batch of up to 10000 URLs using the following methods:

  1. Submit a single URL by sending a request with the URL and your key, like so:
    https://<searchengine>/indexnow?url=https://www.example.org/product.html&key=88b22a29bc244d619f899d57317cb9fa
    Remember to replace <search engine> with either www.bing.com or yandex.com.
  2. Submit up to 10000 URLs using a POST request, like so:
    ‍
    POST /IndexNow HTTP/1.1
    Content-Type: application/json; charset=utf-8
    Host: <searchengine>
    {  
    “host”: “    www.example.org“,  
    “key”: “88b22a29bc244d619f899d57317cb9fa”,  
    “urlList”: [“    https://www.example.org/url1“, “https://www.example.org/folder/url2“, “https://www.example.org/url3”]
    }

    Again, remember to replace <search engine> with either www.bing.com or yandex.com.

For now, Google doesn’t support IndexNow, so using https://www.google.com/indexnow?url=url-changed&key=your-key will have the same effect as using https://www.google.com/HeyGooglePleaseIndexMyContent.html, meaning – no effect at all.
‍
That being said, sending a request to either Bing or Yandex will result in both search engines being informed:
‍
“Starting November 2021, IndexNow-enabled search engines will share immediately all URLs submitted to all other IndexNow-enabled search engines, so when you notify one, you will notify all search engines.”

Automating URL submission

Although the protocol has just been introduced, its creators must have been working with various software providers behind the curtains, because companies like:

  • Wix,
  • Duda,
  • Akamai, and
  • Cloudflare,

already declared that they will support automatically sending rediscovery requests to IndexNow for their clients.

Moreover, popular websites like GitHub, eBay, or LinkedIn, previously adopted Bing’s URL Submission API and are very likely to move towards IndexNow as well.
‍
Even WordPress has its work cut out for it, as Bing provided the open-source code to support IndexNow to help WordPress quickly adopt the shared protocol.
‍
If you aren’t currently using any of these services, don’t worry! We’ve built a tool that you can use to easily submit a list of URLs or your sitemap to IndexNow.

Why use IndexNow

According to the official announcement:
‍
“Without IndexNow, it can take days to weeks for search engines to discover that the content has changed, as search engines don’t crawl every URL often. With IndexNow, search engines know immediately the URLs that have changed, helping them prioritize crawl for these URLs and thereby limiting organic crawling to discover new content.”
‍
Both Bing and Yandex currently have mechanisms in place that you can use to ask for a recrawl of your content, and it’s the same with Google (although Google doesn’t allow submitting multiple URLs at a time).
‍
None of these mechanisms guarantee that your fresh content gets picked up immediately, but they do improve the chances that search engines will pick it up faster than they normally would.
‍
Currently, the main benefit of using IndexNow is that by notifying one search engine, you simultaneously notify others, too. Plus, you can submit URLs in bulks of up to 10000, which is essential for large sites.
‍
This can become even more appealing if more search engines support this shared protocol in the future. If Google joined Bing and Yandex, it would be a game-changer.

When it’s useful to ask search engines for a recrawl

You can use IndexNow (and all the other URL submission mechanisms) to help new content get indexed faster, but there are multiple other use cases:

  • If you make significant changes to your content and you think it deserves to rank better, asking for a recrawl can lead to quicker positive results.
  • If you have 404 pages that you want to delete from the index, it might happen faster if they quickly get recrawled.
  • If you redirect a page, you can ask for a recrawl to see your redirect take effect sooner.

Still, most importantly, “Search engines will generally prioritize crawling URLs submitted via IndexNow versus other URLs they know.” Using IndexNow won’t make Bing or Yandex crawl more of your site, but it can help you influence what gets crawled with a priority.
‍

Are indexing APIs the future of search engines?

As it stands right now, IndexNow is not a major revolution in search, although it’s great to see search engines work together to build something that may improve the web for everyone.
‍
But I think this initiative points to a fact of utmost importance: the existing content discovery process that major search engines share is, at least to an extent, subject to change.
‍
The current process is as follows: search engines look for new URLs to visit on websites they already know, following on-page links and sitemaps. Then they make a list of URLs that’s prioritized based on various factors – they’re mostly confidential but a good rule of thumb is that consistent, great quality, and popularity of your domain will keep the search engines coming back. 
‍
What this process entails is the constant crawl activity, which search engines need to expand as the web grows exponentially. They have little choice – there’s great, relevant content popping up everywhere and that’s on top of existing popular websites which need regular recrawling to keep the search index as fresh as possible.
‍
What if some of the process of discovering new content could be replaced by a mechanism that would allow site owners to proactively inform search engines when they make changes to their websites? 
‍
If this process turned out to be a good alternative, search engines could potentially save resources spent on crawling.
‍
But crawling itself isn’t the main problem with maintaining a large search index of fresh content.
‍

Rendering is expensive

Crawling billions upon billions of pages is definitely a challenge, but it’s rendering them to fully see their contents that’s absurdly resource-consuming.
‍
Google is definitely ahead of the competition when it comes to its ability to render and analyze web content on a scale. While others are catching up, smaller search engines may never be able to spend enough resources on rendering to truly compete with Google when it comes to the quality of their search results.
‍
IndexNow could truly make a difference, particularly for smaller search engines, if it allowed multiple companies to join forces and use a single database of rendered pages to build their search indices.
‍

Wrapping up

If you ever suffered from indexing issues with your website, you should follow the discussion around IndexNow very closely.
Time will tell if this initiative will encourage Google to either use IndexNow data to supplement its own search index or come up with its own URL submission solution that’s more scalable than what we currently have in Google Search Console.
‍
Either way, IndexNow an innovative attempt at solving one of the greatest challenges search engines currently face: the abundance of content that needs to be crawled and rendered while an average website gets heavier every year. 
‍
I applaud everyone involved in developing this protocol and I’m looking forward to seeing how it affects the web in the future.