How to integrate IndexNow: instant search engine indexing for websites, SaaS, and APIs

How to integrate IndexNow: instant search engine indexing for websites, SaaS, and APIs#

For decades, the lifecycle of web indexing operated entirely on a pull model: I would publish a new article or update a page on my site and passively wait for search engine robots to eventually visit my sitemap.xml to discover what changed. On large websites or fresh domains, this discovery interval could take anywhere from days to weeks.

The IndexNow protocol fundamentally flips this architecture to a push model. Instead of waiting for crawlers to wander through the server, my own backend proactively tells search engines exactly which URLs were created, updated, or removed in real time.

In this practical guide, I break down how the protocol works, how cryptographic host validation operates, and share production-ready implementations in JavaScript (Node.js), PHP, Python, and CI/CD pipelines.

1. What is IndexNow and why does it matter?#

Originally launched by Microsoft (Bing) and Yandex, IndexNow is an open protocol published under a Creative Commons license. It tackles two major architectural pain points across the modern web:

1. Content discovery latency: The time required for search engines to discover and index fresh pages drops from weeks to mere seconds or minutes. This is vital for breaking technical content, e-commerce pricing updates, and newly deployed SaaS features.
2. Elimination of crawler server overhead: Traditional bots repeatedly scrape static, unmodified pages simply to determine if any content changed. With IndexNow, search engine crawlers only dedicate network bandwidth and server CPU to pages that were actually touched.
3. Mesh Sharing: When submitting a list of URLs to any participating endpoint (such as api.indexnow.org or www.bing.com), the receiving search engine automatically broadcasts the notification across all other member search engines in the network.

Beyond Bing and Yandex, regional search engines (like Seznam and Naver) as well as modern AI answer engines rely on these unified indices to power accurate, real-time citation pipelines.

2. Host-matching key verification#

IndexNow eliminates complex OAuth handshakes or user account credentials. It relies on a straightforward Host-Matching Key verification system.

Here is the exact verification workflow:

1. Generate a key: Create a random alphanumeric key in hexadecimal format between 8 and 128 characters long (32 lowercase hex characters is the industry standard).
2. Host the verification file: Place a plain text file named {key}.txt at the public root of your domain (e.g., https://myproject.com/2cd1cb62283949fbb51a9f02e642cb7d.txt).
3. File content: The file must contain only the key itself, with no trailing spaces or extra markup.
4. Validation and queuing: When your system submits URLs along with the key, the search engine makes an automated HTTP GET verification to your host to ensure the file exists and its content matches the submitted key. Once confirmed, ownership is proven and the submitted URLs are routed directly to high-priority indexing queues.

To generate a key directly in the Linux terminal:

# Generate a clean 32-character hexadecimal key
openssl rand -hex 16
# Example output: 2cd1cb62283949fbb51a9f02e642cb7d

# Create the key verification file directly in the web root
echo -n "2cd1cb62283949fbb51a9f02e642cb7d" > 2cd1cb62283949fbb51a9f02e642cb7d.txt

3. Protocol API formats#

IndexNow supports two HTTP interfaces:

Option 1: Single URL (HTTP GET)#

Best suited for small personal blogs, one-off updates, or quick verification using a browser or terminal:

https://api.indexnow.org/indexnow?url=https://myproject.com/new-article.html&key=YOUR_KEY_HERE

Option 2: Batch Submission (HTTP POST JSON)#

The primary method for automated SaaS backends, headless CMS platforms, and build pipelines. It allows submitting from 1 up to 10,000 URLs in a single request:

POST /indexnow HTTP/1.1
Host: api.indexnow.org
Content-Type: application/json; charset=utf-8

{
  "host": "myproject.com",
  "key": "2cd1cb62283949fbb51a9f02e642cb7d",
  "keyLocation": "https://myproject.com/2cd1cb62283949fbb51a9f02e642cb7d.txt",
  "urlList": [
    "https://myproject.com/page-1.html",
    "https://myproject.com/page-2.html",
    "https://myproject.com/product-xyz.html"
  ]
}

The keyLocation parameter is optional if the key file lives at the root of the domain, but including it explicitly avoids any DNS or reverse-proxy resolution ambiguity.

4. Implementation in Node.js / JavaScript#

For modern Node.js environments (Express, Fastify, Next.js or static site generator post-build scripts), I handle submissions using the native fetch API:

// indexnow-submit.js
import fs from "node:fs";

const INDEXNOW_API = "https://api.indexnow.org/indexnow";
const HOST = "myproject.com";
const KEY = "2cd1cb62283949fbb51a9f02e642cb7d";

/**
 * Submit URLs to the IndexNow protocol
 * @param {string[]} urls - Full URLs to index
 */
export async function submitToIndexNow(urls) {
  if (!urls || urls.length === 0) {
    console.log("No URLs provided for submission.");
    return { ok: false, error: "Empty URL list" };
  }

  // The protocol accepts up to 10,000 URLs per batch
  const batch = urls.slice(0, 10000);

  const payload = {
    host: HOST,
    key: KEY,
    keyLocation: `https://${HOST}/${KEY}.txt`,
    urlList: batch
  };

  try {
    const response = await fetch(INDEXNOW_API, {
      method: "POST",
      headers: {
        "Content-Type": "application/json; charset=utf-8",
        "User-Agent": "IndexNow-Client/1.0"
      },
      body: JSON.stringify(payload)
    });

    if (response.status === 200) {
      console.log(`[IndexNow] Success: ${batch.length} URLs submitted directly.`);
      return { ok: true, status: 200 };
    } else if (response.status === 202) {
      console.log(`[IndexNow] Accepted: ${batch.length} URLs queued for indexing.`);
      return { ok: true, status: 202 };
    } else {
      const errorText = await response.text();
      console.error(`[IndexNow] HTTP Error ${response.status}:`, errorText);
      return { ok: false, status: response.status, error: errorText };
    }
  } catch (err) {
    console.error("[IndexNow] Network Error:", err.message);
    return { ok: false, error: err.message };
  }
}

// Example usage:
// submitToIndexNow(["https://myproject.com/blog/new-article.html"]);

5. Implementation in PHP#

For standard PHP applications, WordPress hooks, Laravel jobs, or custom APIs, I implement a dedicated notifier class wrapping cURL:

<?php
// IndexNowNotifier.php

class IndexNowNotifier
{
    private string $host;
    private string $key;
    private string $endpoint;

    public function __construct(string $host, string $key, string $endpoint = 'https://api.indexnow.org/indexnow')
    {
        $this->host = $host;
        $this->key = $key;
        $this->endpoint = $endpoint;
    }

    /**
     * Submit a list of URLs to IndexNow
     * @param array<string> $urls
     * @return array
     */
    public function notify(array $urls): array
    {
        if (empty($urls)) {
            return ['ok' => false, 'error' => 'URL list cannot be empty'];
        }

        $payload = [
            'host'        => $this->host,
            'key'         => $this->key,
            'keyLocation' => "https://{$this->host}/{$this->key}.txt",
            'urlList'     => array_values(array_slice($urls, 0, 10000))
        ];

        $jsonPayload = json_encode($payload, JSON_UNESCAPED_SLASHES | JSON_UNESCAPED_UNICODE);

        $ch = curl_init($this->endpoint);
        curl_setopt_array($ch, [
            CURLOPT_POST           => true,
            CURLOPT_POSTFIELDS     => $jsonPayload,
            CURLOPT_RETURNTRANSFER => true,
            CURLOPT_TIMEOUT        => 10,
            CURLOPT_HTTPHEADER     => [
                'Content-Type: application/json; charset=utf-8',
                'Content-Length: ' . strlen($jsonPayload),
                'User-Agent: IndexNow-PHP/1.0'
            ]
        ]);

        $response = curl_exec($ch);
        $httpCode = curl_getinfo($ch, CURLINFO_HTTP_CODE);
        $curlError = curl_error($ch);
        curl_close($ch);

        if ($curlError !== '') {
            return ['ok' => false, 'error' => $curlError];
        }

        if ($httpCode === 200 || $httpCode === 202) {
            return ['ok' => true, 'code' => $httpCode, 'count' => count($payload['urlList'])];
        }

        return [
            'ok'    => false,
            'code'  => $httpCode,
            'error' => $response ?: "HTTP Error {$httpCode}"
        ];
    }
}

// Example usage on content update:
// $notifier = new IndexNowNotifier('myproject.com', '2cd1cb62283949fbb51a9f02e642cb7d');
// $result = $notifier->notify(['https://myproject.com/blog/sample-post.html']);

6. WordPress integration: Rank Math and Official Plugin#

Not every deployment requires custom code. If you manage websites, blogs, or WooCommerce stores built on WordPress, two proven turnkey options exist to integrate IndexNow without manual scripting:

Option A: Rank Math SEO (Instant Indexing Module)#

Rank Math was among the earliest SEO suites to build native IndexNow support through its dedicated Instant Indexing module:

1. Enable the Module: In the WordPress dashboard, navigate to Rank Math > Dashboard and toggle on Instant Indexing.
2. Automated API Key Provisioning: Under the module settings, Rank Math automatically generates and hosts the domain verification key dynamically, eliminating the need to upload text files via FTP or SSH manually.
3. Post Type Targeting: You can configure exactly which content types trigger automatic pings upon publishing or editing (Posts, Pages, WooCommerce Products, or Custom Post Types).
4. Manual and Batch Submissions: Rank Math includes a built-in console where you can paste up to 100 URLs at once for batch submission, or trigger single-click indexing directly from the WordPress posts table.
5. Submission Audit Log: The plugin records timestamped logs displaying submitted URLs and the corresponding HTTP status codes returned by the IndexNow endpoints.

Option B: Official Microsoft IndexNow Plugin#

For environments running lightweight setups without an all-in-one SEO suite:

• Microsoft provides the free official IndexNow plugin on the WordPress repository (wordpress.org/plugins/indexnow/).
• Upon activation, the plugin provisions the host validation key automatically and hooks into WordPress core actions (save_post, transition_post_status, and deletions).
• It monitors content updates and 301 redirects, dispatching background pings whenever content is published, updated, or purged.

7. Implementation in Python#

For Python applications using Django, FastAPI, or task workers:

# indexnow.py
import requests

def notify_indexnow(host: str, key: str, urls: list[str]) -> dict:
    if not urls:
        return {"ok": False, "error": "No URLs provided"}

    endpoint = "https://api.indexnow.org/indexnow"
    payload = {
        "host": host,
        "key": key,
        "keyLocation": f"https://{host}/{key}.txt",
        "urlList": urls[:10000]
    }

    try:
        response = requests.post(
            endpoint,
            json=payload,
            headers={"Content-Type": "application/json; charset=utf-8"},
            timeout=10
        )

        if response.status_code in (200, 202):
            return {"ok": True, "status_code": response.status_code, "count": len(urls)}
        
        return {
            "ok": False,
            "status_code": response.status_code,
            "error": response.text
        }
    except requests.RequestException as e:
        return {"ok": False, "error": str(e)}

# Example:
# res = notify_indexnow("myproject.com", "2cd1cb62283949fbb51a9f02e642cb7d", [
#     "https://myproject.com/services/new-service.html"
# ])
# print(res)

8. CI/CD automation with GitHub Actions and cURL#

For static sites built and deployed via CI/CD, notifying search engines automatically upon deployment ensures zero lag between commit merges and indexing:

# .github/workflows/deploy-and-index.yml
name: Deploy & Notify IndexNow

on:
  push:
    branches: [ main ]

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build static site
        run: npm run build

      - name: Broadcast URLs to IndexNow
        env:
          INDEXNOW_KEY: ${{ secrets.INDEXNOW_KEY }}
        run: |
          curl -s -X POST "https://api.indexnow.org/indexnow" \
            -H "Content-Type: application/json; charset=utf-8" \
            -d '{
              "host": "myproject.com",
              "key": "'"$INDEXNOW_KEY"'",
              "keyLocation": "https://myproject.com/'"$INDEXNOW_KEY"'.txt",
              "urlList": [
                "https://myproject.com/"
              ]
            }'

9. IndexNow HTTP response codes#

When debugging or monitoring production logs, keep this reference handy:

10. Production best practices#

1. Submit canonical, clean URLs only: Never send URLs with noindex headers, intermediate 301 redirects, or robots.txt disallows. Send only final canonical URLs.
2. Strip marketing parameters: Strip all tracking query strings (?utm_, ?fbclid=) before submitting to the endpoint.
3. Queue and batch events: Do not fire individual HTTP requests on every single database record update. Collect touched paths in a cache or queue and broadcast a consolidated batch every 5 to 10 minutes.
4. Notify on content deletion (404/410): When a page is retired, ping the URL to IndexNow so search engines remove the defunct listing from SERPs rapidly, reducing user bounce rates from stale search results.

Implementing IndexNow is one of the highest-leverage technical SEO enhancements available: setup takes under an hour and shifts content distribution from passive waiting to instantaneous, event-driven search indexing.

See also

Development Upgrading Vite in Laravel: from kernel inotify to manifest.json

Development Debugging WhatsApp cloud API: how to resolve 131049 and 130472 errors

Development Bun, Node.js, and PM2 in production: when to use, when to avoid, and architectural trade-offs

This post is licensed under CC BY-NC.