Stop waiting for crawlers. IndexNow lets you notify Bing, Yandex, and other engines the moment you publish or update a page. This guide covers server-side setup, API key management, and how it compares to sitemaps and manual pinging.
You push a sitemap. Then what? You wait. Googlebot might show up in 3 hours or 3 days. IndexNow flips this model: you push the URL, the engine indexes it. No queue, no guesswork. The protocol is a simple HTTP GET with one parameter: the URL you want indexed. That's it. No authentication, no OAuth, no payload.
Here's the catch many miss: IndexNow is not a replacement for sitemaps. Sitemaps are still the best way to signal your site's full structure. IndexNow is for high-urgency signals — breaking news, price drops, updated stock, new blog posts. Use both. A common situation we see is teams setting up IndexNow and then removing their sitemap submission. Bad move. Sitemaps handle bulk discovery; IndexNow handles instant notification. Different tools, same goal.
| Feature | IndexNow Protocol | XML Sitemap | Search Console API Ping | Verdict / Best Fit |
|---|---|---|---|---|
| Trigger What initiates submission? | Manual or automated HTTP GET request | Crawler discovers via robots.txt or manual submission | Google Search Console API call with auth | IndexNow wins for urgency; sitemap for discovery |
| Latency Time to index | < 60 seconds typically | Hours to days | Minutes to hours depending on crawl capacity | IndexNow is fastest for single URLs |
| Bulk capacity URLs per call | Up to 10,000 URLs per POST body | Unlimited (but file size limit 50 MB) | Single URL per call (batch via loop) | Sitemap for bulk; IndexNow for selective pushes |
| Auth complexity Setup overhead | One API key (text file in root) | None (just file placement) | OAuth 2.0 + scope configuration | IndexNow is simplest for server-side automation |
| Error handling Failure modes | HTTP 200 = success; 4xx/5xx = failure; no retry logic built-in | No real-time feedback; errors found in crawl logs later | Returns JSON error with details; supports retry | API ping has best diagnostics; IndexNow is silent |
| Hidden risk Operational failure | Key file missing, wrong domain in URL, blocked by robots.txt | Sitemap not referenced in robots.txt, compressed with wrong encoding | Quota exceeded (200 URLs/day for free tier), token expiry | IndexNow fails silently — you must log responses |
You run an e-commerce site with 50 product pages updated after a price refresh. Here's the sequence:
Step 1: Generate API key: 7b3e4f2a-1c8d-4e6f-9a0b-2c3d4e5f6a7b. Place the file at root of domain.
Step 2: Build the batch POST body (JSON):{
"host": "www.yourstore.com",
"key": "7b3e4f2a-1c8d-4e6f-9a0b-2c3d4e5f6a7b",
"urlList": [
"https://www.yourstore.com/product/1",
"https://www.yourstore.com/product/2",
... 50 URLs total
]
}
Step 3: Send POST to https://api.indexnow.org/indexnow with Content-Type: application/json.
Step 4: Check response. You get 200 OK. But 3 URLs return 400 because they contain trailing slashes that conflict with the host definition. Edge case: URL normalization matters. Strip trailing slashes from all URLs in the list. Resubmit the corrected 3.
Step 5: Verify indexing. Check Bing Webmaster Tools 'URL Inspection' for one of the corrected URLs. It should show 'indexed within 5 minutes'. If not, your key file might be cached by a CDN — flush the edge cache.
Create UUID and upload as .txt file to domain root
Gather newly published or updated URLs (max 10,000 per batch)
GET for single URL, POST with JSON body for batch
200 = queued. 4xx/5xx = error. Log all responses.
Use <a href='https://developers.google.com/search/docs/monitor-debug/search-console-start'>Google Search Console</a> URL inspection tool to confirm indexing
Check Bing Webmaster Tools 'IndexNow stats' for success rate
Your key file must be world-readable but not in a subdirectory. If you use Cloudflare or a CDN, the key file must bypass the cache — otherwise the search engine might get a stale response. I've seen teams spend hours debugging only to find their 404 page was being served for the key file path. Another classic: submitting URLs that return 4xx or 5xx themselves. IndexNow doesn't check page health — it just queues the URL. If your server is down when the crawler arrives, the URL falls out of the index. You must resubmit after the server recovers.
Duplicate submissions aren't harmful but they waste your quota. Filter your URL list to remove duplicates before the POST body. A simple sort -u on your list before curl can save you 30% of your daily limit. Also note: IndexNow honors robots.txt disallow rules. If you block a path, the URL will be ignored even if you ping it. Always check robots.txt before submission.
| Option | What happens | Verdict |
|---|---|---|
| You publish 1 critical page (product launch, breaking news) | Use IndexNow single URL GET | Instant notification. 1 second. |
| You update 500 product prices in bulk | Use IndexNow batch POST (up to 10k per call) | Fast and efficient, but monitor quota. |
| You add 15,000 new archive pages | Use XML Sitemap submission | Bulk discovery, not urgency. IndexNow limit is 10k per day. |
| You need indexing confirmation with error details | Use Google Search Console API Ping | Better diagnostics, but OAuth overhead. |
API key file is placed at the exact root path (no subfolder)
Key file is accessible without authentication (HTTP 200, no redirect)
Key file is NOT blocked by robots.txt or .htaccess
CDN or proxy cache is bypassed for the key file path
URLs in the submission list are absolute and use HTTPS
URLs are not duplicated in a single batch (use unique filter)
You have a logging mechanism to capture HTTP responses
You have a retry mechanism for failed URLs (429, 5xx)
Generate a unique API key per domain. Each domain needs its own key file at its root. Automate via a script that loops over a list of domains, generates a UUID per domain, uploads the file via FTP or API, and then submits URLs using the corresponding key. Monitor each domain's key file accessibility separately.
Yes, but you need access to the external site's server to place the key file. If you don't have that access, you cannot use IndexNow for that domain. Instead, ask the site owner to submit the URL themselves or use a shared key if the platform supports it (rare).
IndexNow will queue the URL, but when the crawler arrives and gets a 404, the page will not be indexed. You must ensure the page returns 200 when the crawler hits it. Best practice: submit the URL only after the page is live and verified.
Yes. Use a POST request with a JSON body containing up to 10,000 URLs. The endpoint is https://api.indexnow.org/indexnow. Set Content-Type to application/json. Each URL must be absolute and point to the same host as the key file. Bulk is the preferred method for efficiency.
HTTP 400: malformed URL or invalid key format. HTTP 403: key file not found or blocked. HTTP 422: URL host does not match key file domain. HTTP 429: rate limit exceeded — wait and retry. HTTP 500: server error — retry after a few minutes. Always log the full response body for debugging.
Yes: 1) Generate UUID key. 2) Upload key file to domain root. 3) Verify file accessible via browser. 4) Test single URL submission. 5) Check response code. 6) Monitor Bing Webmaster Tools for success rate. 7) Automate with cron or webhook. 8) Set up logging. 9) Test with a URL that you can later inspect in search console.
First, check the key file returns 200 and is not cached. Use curl -I https://yourdomain.com/YOURKEY.txt. Second, verify the submitted URL is not blocked by robots.txt. Third, check Bing Webmaster Tools IndexNow report for rejected submissions. Fourth, check server logs for the IndexNow crawler user-agent hitting your URLs.
Most engines (Bing, Yandex) allow up to 10,000 URLs per key per day. Submitting more will result in HTTP 429 responses. The limit resets at midnight UTC. If you need more, split across multiple keys or use sitemaps for bulk discovery and IndexNow only for high-urgency pages.
Yes, any URL that returns a crawlable resource can be submitted. However, IndexNow is primarily designed for HTML pages. For PDFs, ensure they are not blocked by robots.txt and have a useful title and description. Images are better handled via XML sitemaps with image extensions.
Add a post-deployment step in your pipeline (e.g., GitHub Actions, Jenkins) that runs a script to collect all changed URLs from the build artifact, deduplicate them, and send a POST request to the IndexNow API. Store the API key as a secret environment variable. Log the response and fail the build if more than 10% of URLs return errors.
Quick calculator. Put in the expected monthly value of a page or link batch and the natural waiting time.