Discovered but not indexed: what to do when Google leaves you waiting

"Discovered - currently not indexed": why Google delays the crawl decision#

If you manage a website, monitor access logs, and follow the Google Search Console (GSC), you've definitely come across this persistent warning: "Discovered - currently not indexed".

Currently, I am facing this scenario with a batch of 200 URLs. In the dashboard, validation is marked as "Started," but Googlebot has not yet performed the definitive crawl. For an Infrastructure Engineer, this isn't just about "SEO"; it's a matter of resource visibility and discovery efficiency.

Basically, what the algorithm is saying is: "I know your endpoint exists, but I haven't decided yet if it's worth spending CPU time and bandwidth reading it right now".

1) The "discovered but not indexed" scenario and validation cycle#

The "Discovered - currently not indexed" status represents one of the most common and frustrating scenarios in the Google Search Console ecosystem. This warning indicates that Googlebot has found the URL paths (usually by scanning XML Sitemaps or through internal links), but chose to postpone the processing and rendering stage of the content. Unlike an explicit technical failure, such as a 404 or 500 error, this status signals a deliberate decision by the crawler to wait for optimal server or content conditions before indexing the resource.

2) Sitemap and robots.txt: the basic crawl flow#

The first line of audit must focus on the core instruction files for search engines. A syntax error in these files can ruin the entire indexability chain of the site.

1. Validate the presence and HTTP accessibility of your project's sitemap:

curl -I https://perciocastelo.com.br/sitemap.xml

1. Make sure that the URLs with indexing issues are listed and structured in the sitemap.xml file:

curl -s https://perciocastelo.com.br/sitemap.xml | grep -c "discovered-but-not-indexed-what-to-do-when-google-leaves-you-waiting"

1. Audit the policies and blocks defined in the robots.txt file:

curl -s https://perciocastelo.com.br/robots.txt

• Make sure that no Disallow instructions are accidentally blocking the directories of the problematic URLs.
• Confirm that the absolute URL of the sitemap.xml is specified in the footer of robots.txt for automatic discovery.

3) Canonicity and duplicate content prevention#

Googlebot will back off if it suspects that the new URLs are just copies or variations of already existing paths (semantic duplicity). To ensure indexing integrity:

1. Make sure the canonical tag is inserted in the <head> of the page and points exactly to the corresponding self-referential URL:

curl -s https://example.com/blog/en/article-slug.html | grep -i 'rel="canonical"'

1. Compare MD5 hash signatures between similar pages to ensure the content is not identical:

curl -s https://example.com/original-page | md5sum
curl -s https://example.com/suspected-duplicate-page | md5sum

If both signatures return the exact same hexadecimal string, Googlebot will detect structural duplication and push the crawling of the duplicate page to the end of the queue.

4) Internationalization and hreflang validation#

If you maintain a multilingual blog or routes with different geographical locations, inconsistent hreflang tags can cause disorientation and rejection of the pages in the bot's workflow:

1. Make sure the alternate language tags are present in the header of the HTML code:

curl -s https://example.com/blog/en/article-slug.html | grep -i "hreflang"

1. Validate that the mappings are reciprocal. If the Portuguese version points to the English version via hreflang="en", the English version must return the corresponding link pointing back to the Portuguese route via hreflang="pt-br", containing absolute links without intermediate redirects.

5) Data schemas and rich snippets#

The injection of structured data schemas (Schema Markup) helps the algorithm understand the page's type (such as posts, products, or tutorials) even before deep indexing:

1. Validate the physical presence of JSON-LD blocks in the HTML content returned by the server:

curl -s https://example.com/blog/en/article-slug.html | grep -o 'application/ld+json'

1. Extract the structured content of the schema to check for required properties (such as headline, author, datePublished, or publisher):

curl -s https://example.com/blog/en/article-slug.html | grep -A20 "application/ld+json"

6) Mobile-first indexing and viewport#

Google uses a mobile-first indexing model. If the layout breaks on smartphones or does not have adequate responsive support, it will not be indexed.

1. Test your endpoint's mobile optimization by querying the official PageSpeed Insights API (using the mobile category):

curl "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://perciocastelo.com.br/&category=MOBILE"

1. Check if the essential meta viewport tag is declared at the top of the HTML <head>:

curl -s https://perciocastelo.com.br/ | grep -i "viewport"

(The recommended default configuration is <meta name="viewport" content="width=device-width, initial-scale=1">).

7) Performance and core web vitals (CWV)#

Slow servers or those with high TTFB (Time to First Byte) force Googlebot to slow down the crawl rate to avoid harming the active users of the server.

1. Test your site's performance locally by running a Lighthouse scan via CLI to audit LCP (Largest Contentful Paint), CLS (Cumulative Layout Shift) metrics, and backend speed:

npx lighthouse https://perciocastelo.com.br/ --output=html --output-path=/tmp/lighthouse-report.html

1. Inspect the initial HTTP response time (TTFB) directly in the terminal with cURL:

curl -o /dev/null -s -w 'Initial Response Time (TTFB): %{time_starttransfer}s\n' https://perciocastelo.com.br/

TTFB values above 0.6s (600ms) are classified as slow and act as a trigger for crawlers to back off.

8) Transport security (https/ssl) and mixed content#

Google prioritizes secure browsing. Sites without valid SSL certificates or containing mixed insecure links (Mixed Content) suffer severe crawling restrictions:

1. Audit the integrity and expiration date of the server's active SSL certificate:

echo | openssl s_client -connect perciocastelo.com.br:443 2>/dev/null | openssl x509 -noout -dates

1. Validate that default navigation in HTTP (port 80) is consistently redirected with a 301 Moved Permanently status code to HTTPS (port 443):

curl -I http://perciocastelo.com.br/

1. Scan the page to ensure there are no absolute internal links or assets (images, scripts, CSS) loaded via the insecure http:// protocol:

curl -s https://perciocastelo.com.br/ | grep -o 'http://'

9) Forensic audit of internal links (link equity)#

A page without internal links pointing to it is labeled as an orphan page. Since Googlebot discovers paths by navigating links, orphan pages receive minimal priority and remain in indexing limbo.

1. Extract and list all absolute internal hyperlinks from a specific source page:

curl -s https://example.com/blog/ | grep -o 'href="[^"]*"' | grep 'example.com'

1. Verify that the URL pending indexing is referenced in the code of other active high-authority posts:

curl -s https://example.com/blog/older-article.html | grep -c "discovered-but-not-indexed"

Ensure new URLs receive at least 3 to 5 contextual links from posts that are already indexed and receive high organic traffic.

10) Triage of server access logs (googlebot)#

If you want to know if Googlebot actually tried to access your URLs or if it gave up along the way, inspect the Nginx access log files in /var/log/nginx/access.log:

# Monitor active Googlebot requests in real-time
tail -f /var/log/nginx/access.log | grep -i "googlebot"

# Filter access attempts on the problematic URLs
grep -i "discovered-but-not-indexed" /var/log/nginx/access.log | grep -i "googlebot"

Verify the HTTP return code recorded in the log:

• 200 OK Status: The bot successfully accessed and read the content. The indexing issue is purely semantic quality or crawl budget, not technical.
• 429 or 503 Status: The server is limiting calls or presented drops in performance. The robots.txt or Nginx configuration must be adjusted.

11) Contingency plan and corrective actions#

To systematically resolve the "Discovered - currently not indexed" status in batches of URLs, follow the operational runbook below:

1. Fix Redirects: Remove any temporary redirects (302) or broken URLs that may have occurred during the discovery cycle.
2. Resubmit Sitemaps: Access the Google Search Console panel -> Sitemaps and resubmit the updated main sitemap.
3. Distribute Link Equity: Temporarily insert the affected URLs into the global sidebar or footer of the site to ensure a daily flow of crawls.
4. DNS Verification: Ensure there are no lookup failures in the DNS server (such as timeouts) by monitoring with the dig command:

dig perciocastelo.com.br +trace

12) Complete indexation validation checklist#

Make sure to validate all points before requesting a new verification in Google Search Console:

• [ ] XML structure and presence of the sitemap.xml file validated.
• [ ] Confirmed that the robots.txt file does not block new directories.
• [ ] Self-referential canonical tag validated on all affected pages.
• [ ] Checked via md5sum that there is no content duplication between pages.
• [ ] Alternate hreflang language tags configured reciprocally in multilingual environments.
• [ ] Valid JSON-LD Schema markup inserted in the HTML.
• [ ] Responsive meta viewport configured and Mobile-Friendliness validated via API.
• [ ] Core Web Vitals metrics (LCP, CLS, TTFB) audited with Lighthouse CLI.
• [ ] Forced 301 redirect from HTTP to HTTPS active and free of mixed content (Mixed Content).
• [ ] Mapped and injected contextual internal links (at least 3 references for each link).
• [ ] Server access logs /var/log/nginx/access.log monitored, confirming Googlebot requests with code 200.

13) Indexability risk matrix#

The following table consolidates indexing threats, severity, and mitigating actions:

Technical conclusion#

The "Discovered - currently not indexed" status should not be treated as a conventional SEO issue, but as a resource prioritization diagnostic. Server speed optimization, intact instruction files, and a solid graph of internal links and canonicity signals tell the algorithm unequivocally that the site's content is valuable and deserves priority indexing.

See also

Infrastructure Anatomy of a phantom subsystem: debugging Bluetooth on Linux

Infrastructure Perfect storm on legacy hardware

Infrastructure Container engineering: replacing WSL with an isolated Podman and Distrobox workflow on Linux Mint

This post is licensed under CC BY-NC.