How to integrate IndexNow: instant search engine indexing for websites, SaaS, and APIs

Until recently, publishing content or running a web application meant waiting on crawler schedules. You shipped a new article or added a route, refreshed your sitemap.xml, and waited days (sometimes weeks) for search engine bots to visit. To make matters worse, Google permanently shut down its public sitemap ping endpoint in late 2023, making it clear that passive pull-based discovery has clear limits.

The IndexNow protocol flips this around into a push architecture. Instead of waiting for a crawler to eventually stumble upon server updates, my own application sends an immediate HTTP notification to participating search engines: "this URL was just created or modified, come crawl it".

In this guide, I break down how the protocol functions under the hood, how domain key verification works, and how I built clean implementations in Node.js, PHP, Python, and CI/CD pipelines.

1. What is IndexNow and who actually uses it?#

Created by Microsoft (Bing) and Yandex as an open protocol under a Creative Commons license, IndexNow is lean and purpose-built. It resolves two core bottlenecks in web search infrastructure:

1. Near-instant discovery: The delay between publishing an article or product page and the crawler's visit drops from days down to seconds or minutes.
2. Lower server bandwidth and CPU usage: Traditional bots repeatedly scrape static, unchanged pages just to verify whether anything moved. With push indexing, crawlers only hit URLs where real modifications occurred.

Beyond solving these two bottlenecks, the protocol features an architectural advantage: mesh sharing — when submitting URLs to any participating endpoint (like api.indexnow.org or www.bing.com), the receiving engine automatically broadcasts the notification to all other partner search engines in the network (Yandex, Seznam, Naver).

Where does AI fit into this? AI engines that synthesize real-time answers (like Microsoft Copilot and tools querying Bing's search index) consume this fresh data feed. If Bing indexes your newly deployed page within minutes, AI assistants can cite your content immediately.

2. Host-matching key verification: simple, credential-free ownership#

IndexNow avoids cumbersome OAuth handshakes, dedicated developer accounts, or client-side tokens. It confirms domain ownership via HTTP through a Host-Matching Key:

1. Generate a key: A random hexadecimal string between 8 and 128 characters long (32 lowercase hex characters is standard).
2. Host the verification file: Place a plain text file named {key}.txt at the public document root of your domain (for example: https://myproject.com/2cd1cb62283949fbb51a9f02e642cb7d.txt).
3. File contents: The file must contain only the key string itself, with no whitespace, newlines, or HTML tags.
4. Validation check: When the search engine receives a notification with your URLs, its crawler makes a quick HTTP GET request to verify the file on your host. If the file exists and its content matches the submitted key, ownership is verified and the URLs enter the prioritized indexing queue.

To generate the key and verification file directly in the Linux terminal:

# Generate a clean 32-character hexadecimal key
openssl rand -hex 16
# Example output: 2cd1cb62283949fbb51a9f02e642cb7d

# Save the file to your web server document root
echo -n "2cd1cb62283949fbb51a9f02e642cb7d" > 2cd1cb62283949fbb51a9f02e642cb7d.txt

3. Protocol API formats#

The IndexNow endpoint supports two straightforward submission interfaces:

Method 1: Single URL via HTTP GET#

Useful for terminal health checks with curl or quick manual tests:

https://api.indexnow.org/indexnow?url=https://myproject.com/new-article.html&key=YOUR_KEY_HERE

Method 2: Batch Submission via HTTP POST (JSON)#

The production standard for application code and CI/CD pipelines, supporting anywhere from 1 up to 10,000 URLs per request:

POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "myproject.com",
  "key": "2cd1cb62283949fbb51a9f02e642cb7d",
  "keyLocation": "https://myproject.com/2cd1cb62283949fbb51a9f02e642cb7d.txt",
  "urlList": [
    "https://myproject.com/blog/article-1.html",
    "https://myproject.com/blog/article-2.html",
    "https://myproject.com/pricing.html"
  ]
}

The keyLocation field is optional if your key file is at the host root, but I always specify it explicitly to eliminate resolution ambiguity across multi-domain and reverse-proxy architectures.

4. Implementation in Node.js / JavaScript#

For modern Node.js backends (Express, Fastify, Next.js server actions, or SSG post-build scripts), the built-in fetch API handles submissions cleanly:

// indexnow-submit.js
const INDEXNOW_API = "https://api.indexnow.org/indexnow";
const HOST = "myproject.com";
const KEY = "2cd1cb62283949fbb51a9f02e642cb7d";

/**
 * Submit URLs to the IndexNow protocol
 * @param {string[]} urls - Full URLs to index
 */
export async function submitToIndexNow(urls) {
  if (!urls || urls.length === 0) {
    return { ok: false, error: "No URLs provided." };
  }

  const batch = urls.slice(0, 10000);

  const payload = {
    host: HOST,
    key: KEY,
    keyLocation: `https://${HOST}/${KEY}.txt`,
    urlList: batch
  };

  try {
    const response = await fetch(INDEXNOW_API, {
      method: "POST",
      headers: {
        "Content-Type": "application/json; charset=utf-8",
        "User-Agent": "IndexNow-Client/1.0"
      },
      body: JSON.stringify(payload)
    });

    if (response.status === 200 || response.status === 202) {
      console.log(`[IndexNow] ${batch.length} URLs submitted successfully (HTTP ${response.status}).`);
      return { ok: true, status: response.status, count: batch.length };
    }

    const errorBody = await response.text();
    console.error(`[IndexNow] HTTP Error ${response.status}:`, errorBody);
    return { ok: false, status: response.status, error: errorBody };
  } catch (err) {
    console.error("[IndexNow] Network Error:", err.message);
    return { ok: false, error: err.message };
  }
}

// Example:
// submitToIndexNow(["https://myproject.com/blog/latest-post.html"]);

5. Implementation in pure PHP#

For PHP applications (custom frameworks, Laravel jobs, or standalone utilities), a lean cURL class gets the job done without external dependencies:

<?php
// IndexNowNotifier.php

class IndexNowNotifier
{
    private string $host;
    private string $key;
    private string $endpoint;

    public function __construct(string $host, string $key, string $endpoint = 'https://api.indexnow.org/indexnow')
    {
        $this->host = $host;
        $this->key = $key;
        $this->endpoint = $endpoint;
    }

    public function notify(array $urls): array
    {
        if (empty($urls)) {
            return ['ok' => false, 'error' => 'URL list cannot be empty.'];
        }

        $payload = [
            'host'        => $this->host,
            'key'         => $this->key,
            'keyLocation' => "https://{$this->host}/{$this->key}.txt",
            'urlList'     => array_values(array_slice($urls, 0, 10000))
        ];

        $jsonPayload = json_encode($payload, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);

        $ch = curl_init($this->endpoint);
        curl_setopt_array($ch, [
            CURLOPT_POST           => true,
            CURLOPT_POSTFIELDS     => $jsonPayload,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 10,
            CURLOPT_HTTPHEADER     => [
                'Content-Type: application/json; charset=utf-8',
                'Content-Length: ' . strlen($jsonPayload),
                'User-Agent: IndexNow-PHP/1.0'
            ]
        ]);

        $response = curl_exec($ch);
        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $curlError = curl_error($ch);
        curl_close($ch);

        if ($curlError !== '') {
            return ['ok' => false, 'error' => $curlError];
        }

        if ($httpCode === 200 || $httpCode === 202) {
            return ['ok' => true, 'code' => $httpCode, 'count' => count($payload['urlList'])];
        }

        return [
            'ok'    => false,
            'code'  => $httpCode,
            'error' => $response ?: "HTTP Error {$httpCode}"
        ];
    }
}

// Example usage:
// $client = new IndexNowNotifier('myproject.com', '2cd1cb62283949fbb51a9f02e642cb7d');
// $status = $client->notify(['https://myproject.com/blog/my-article.html']);

6. WordPress: Enabling IndexNow with Rank Math or the Official Plugin#

If you manage websites, content portals, or WooCommerce stores on WordPress, writing custom code is unnecessary. Two reliable methods handle IndexNow out of the box:

Option A: Rank Math SEO (Instant Indexing Module)#

If you already use the Rank Math suite, native IndexNow support is built right into the core plugin:

1. Enable the module: In your WordPress dashboard, navigate to Rank Math > Dashboard and toggle on Instant Indexing.
2. Automatic key management: Under Instant Indexing settings, Rank Math provisions and dynamically serves the domain key without requiring manual FTP or SSH file uploads.
3. Select content types: Choose which post types trigger automatic search engine pings upon publishing or editing (Posts, Pages, Products, or Custom Post Types).
4. Manual and batch submissions: The module provides a submission console where you can paste up to 100 URLs for immediate broadcasting, plus a quick action button inside the WordPress posts table.
5. Submission audit log: Check timestamped logs displaying submitted URLs and the HTTP status codes returned by the engines.

Option B: Microsoft's Official IndexNow Plugin#

If you do not run Rank Math and prefer a dedicated, lightweight tool without extra SEO features:

• Install the free IndexNow plugin provided directly by Microsoft from the WordPress repository (wordpress.org/plugins/indexnow/).
• Once active, it generates the verification key automatically and hooks into WordPress actions (save_post, transition_post_status, and deletions).
• Any time content is created, updated, or removed (including 301 redirects), the plugin dispatches background pings to the IndexNow API.

7. Implementation in Python#

For FastAPI services, Django applications, or data migration scripts:

# indexnow.py
import requests

def notify_indexnow(host: str, key: str, urls: list[str]) -> dict:
    if not urls:
        return {"ok": False, "error": "URL list is empty"}

    endpoint = "https://api.indexnow.org/indexnow"
    batch = urls[:10000]
    payload = {
        "host": host,
        "key": key,
        "keyLocation": f"https://{host}/{key}.txt",
        "urlList": batch
    }

    try:
        response = requests.post(
            endpoint,
            json=payload,
            headers={"Content-Type": "application/json; charset=utf-8"},
            timeout=10
        )

        if response.status_code in (200, 202):
            return {"ok": True, "status_code": response.status_code, "count": len(batch)}
        
        return {
            "ok": False,
            "status_code": response.status_code,
            "error": response.text
        }
    except requests.RequestException as e:
        return {"ok": False, "error": str(e)}

# Example:
# res = notify_indexnow("myproject.com", "2cd1cb62283949fbb51a9f02e642cb7d", [
#     "https://myproject.com/features/new-feature.html"
# ])
# print(res)

8. CI/CD automation with GitHub Actions#

For static site generators (custom builders, Astro, 11ty, or Hugo) deployed to Cloudflare Pages, GitHub Pages, or Vercel, the best time to ping search engines is right at the end of the deployment pipeline:

# .github/workflows/deploy-and-index.yml
name: Deploy & Notify IndexNow

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build static site
        run: npm run build

      - name: Broadcast updated URLs to IndexNow
        env:
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
        run: |
          curl -s -X POST "https://api.indexnow.org/indexnow" \
            -H "Content-Type: application/json; charset=utf-8" \
            -d '{
              "host": "myproject.com",
              "key": "'"$INDEXNOW_KEY"'",
              "keyLocation": "https://myproject.com/'"$INDEXNOW_KEY"'.txt",
              "urlList": [
                "https://myproject.com/"
              ]
            }'

9. IndexNow HTTP status codes#

When inspecting logs or building alert mechanisms, use this reference table:

10. Production best practices#

1. Submit canonical URLs only: Never broadcast URLs with noindex robots tags, intermediate 301 redirects, or robots.txt disallows. Send only final, clean destinations.
2. Strip query parameters: Remove tracking strings such as ?utm_ or ?fbclid= before generating the payload.
3. Batch high-volume updates: If your database updates hundreds of records per minute, do not fire an HTTP call per database write. Queue changed URLs (using Redis or a lightweight message broker) and dispatch a consolidated batch every 5 to 10 minutes.
4. Notify on content removal (404/410): When a page is permanently taken down, submit the URL to IndexNow so search engines purge defunct listings quickly, minimizing 404 bounces for users.

Conclusion: event-driven, friction-free indexing#

In this guide, I covered the end-to-end IndexNow implementation path — from host-matching key verification to automated CI/CD pipelines. The integration cost is minimal and the content discovery acceleration is immediate: it benefits any website, SaaS, or API where fresh content must be discovered in minutes rather than days.

IndexNow won't fix poor content quality or replace technical SEO fundamentals, but it eliminates the friction of passive web crawling. Instead of waiting days for search bots to wander in, your server takes the initiative and notifies search engines the moment updates are live.

See also

Development Upgrading Vite in Laravel: from kernel inotify to manifest.json

SEO & Performance Rank Math Free vs PRO: When Is It Really Worth Upgrading? A Technical Review

Development Debugging WhatsApp cloud API: how to resolve 131049 and 130472 errors

This post is licensed under CC BY-NC.