Complete guide to website txt and well-known files: from ads.txt to security.txt
Back to blog

Complete guide to website txt and well-known files: from ads.txt to security.txt

9/25/2026 · 13 min · Infrastructure

Text files and web root: what actually matters#

When I built my first website, the root directory was simple: an index.html file, a stylesheet, and at most a robots.txt file to keep search engine crawlers out of private folders.

Over time, web architecture quietly accumulated layers of protocol rules. Run any automated SEO auditor today, and you will get persistent warnings claiming your site is broken without ads.txt. Browse technical forums, and people will tell you that researchers cannot reach you without security.txt. With the rapid rise of large language models, brand-new conventions like llms.txt entered the picture.

I wanted to find out what is genuinely necessary, what is merely recommended, and what is pure myth passed down through outdated webmaster tutorials. I tested each standard on my own server, reviewed IANA registries, inspected live production domains with curl, and documented the real-world operational truth behind every file.

Here is the complete guide to all plain text files (.txt, extensionless manifests, and static JSON standards) across the modern web, with live production URLs you can test right now in your browser or terminal.

1) Search engines and indexing: robots, sitemaps, and legacy urllist#

These files talk directly to web search crawlers to establish crawl permissions and provide full content maps.

robots.txt: the classic gatekeeper#

The /robots.txt file is the oldest and most universal crawl directive file on the web. It tells bots (Googlebot, Bingbot, and dozens of others) which paths are open and which should be left alone.

For a massive live example, take a look at Google Brazil robots.txt. The basic pattern adopted by most sites looks like this:

User-agent: *
Disallow: /search
Disallow: /admin/
Disallow: /temp/

Sitemap: https://www.google.com.br/sitemap.xml

Practical verdict: essential for any public website. Even if you want everything indexed, including a pointer to your sitemap speeds up crawler discovery.

sitemap.xml and plain text sitemap.txt#

Most people only associate sitemaps with XML documents like Google Sitemap. Few realize that the official protocol fully supports plain text files: the sitemap.txt.

It contains no XML tags, no headers, and no change dates (lastmod). It is simply a list of absolute URLs, one per line, encoded in UTF-8:

https://perciocastelo.com.br/
https://perciocastelo.com.br/about/
https://perciocastelo.com.br/blog/

You can view public examples in public crawler repositories like englishextra urllist.txt.

Practical verdict: Google Search Console supports .txt sitemaps out of the box. For small static websites, skipping XML generators in favor of a plain text list saves time and effort.

urllist.txt: Yahoo! legacy#

During the mid-2000s, Yahoo! Search / Yahoo! Site Explorer required text-based sitemaps to be named urllist.txt (or compressed as urllist.txt.gz).

Once Yahoo! merged search infrastructure with Bing and standardized on sitemaps.org, the mandatory name disappeared. Today, Google and Bing accept any filename ending in .txt, though urllist.txt still surfaces in older documentation.

robots-staging.txt: pre-production isolation#

On staging or development environments (such as dev. or staging. subdomains), the rule is simple: complete disallow. The /robots.txt file needs only two lines:

User-agent: *
Disallow: /

Practical verdict: a basic safeguard to keep search engines from indexing unfinished work or penalizing your main site for duplicate content.

2) Language models and AI scrapers: llms.txt and the ai.txt myth#

With the arrival of AI chatbots and search agents, new plain text conventions appeared to deliver clean content without HTML layout bloat.

llms.txt: markdown site summaries#

Initiated by Answer.AI, /llms.txt provides a clean Markdown overview placed at the web root or documentation root. Language models waste compute extracting plain text from heavy JavaScript menus, headers, and CSS. This file offers a clean project summary and direct links to essential docs.

Live production examples:

The layout is intentionally simple:

# Anthropic Documentation
> Anthropic is an AI safety and research company that builds reliable, beneficial AI systems.

## Core Documentation
- [Quickstart Guide](https://docs.anthropic.com/en/docs/quickstart): Get started building with Claude.
- [Prompt Engineering](https://docs.anthropic.com/en/docs/build-with-claude/prompt-engineering): Best practices.
- [API Reference](https://docs.anthropic.com/en/api): Complete technical reference.

Practical verdict: if you publish technical tutorials, open-source code, or developer documentation, this is a great addition. It helps coding agents and AI search engines cite your work accurately.

llms-full.txt: concatenated complete documentation#

While llms.txt acts as an outline with links, llms-full.txt packs the entire documentation into a single plain text file. Models with large context windows (like Gemini and Claude) can ingest the full documentation at once without issuing dozens of HTTP fetches.

You can inspect this format at Anthropic llms-full.txt, which aggregates guides, API specs, and SDK references into a multi-megabyte Markdown file.

ai.txt: the proposal that never materialized#

You may have seen blog posts suggesting an /ai.txt file to block AI model training. The reality is simple: ai.txt never became an adopted standard.

None of the major AI labs (OpenAI, Google, Anthropic, Meta) read or respect this file. Projects that tried to host references (like spawning.ai or ai.txt.org) let their domains lapse or abandoned the specifications. Creating this file has zero practical effect.

Publishers looking to control AI scraping today rely on two proven options:

  1. User-agents in traditional robots.txt: News organizations like The New York Times block AI bots directly in their standard robots.txt:
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /
  1. The European tdmrep.json standard: In the EU, publishers reserving text and data mining rights by law implement the W3C manifest at /.well-known/tdmrep.json.

3) Security, cryptography, and email: security.txt, mta-sts, and public keys#

This category covers formal IETF standards (mostly mapped under RFC 8615 in /.well-known/) to publish security channels and protect network traffic.

security.txt: responsible vulnerability reporting#

Standardized under RFC 9116, /.well-known/security.txt solves a straightforward problem: how can security researchers contact you when they find an issue on your website?

Without it, reporters have to guess contact forms or generic social profiles.

Major platforms keep this file updated:

The template I use on my domain includes the core fields:

Contact: mailto:[email protected]
Expires: 2027-12-31T23:59:59.000Z
Preferred-Languages: pt, en
Canonical: https://perciocastelo.com.br/.well-known/security.txt

Practical verdict: the Expires field is mandatory to prevent abandoned contact details. A simple, static file that belongs on any active website.

mta-sts.txt: enforcing TLS for email#

MTA-STS (RFC 8461) protects server-to-server email exchanges against cleartext downgrade attacks over SMTP.

The RFC requires the file to be served on a dedicated subdomain (mta-sts.<domain>) at /.well-known/mta-sts.txt strictly via HTTPS with a valid certificate.

Two live examples:

The policy file specifies MX hosts and mode:

version: STSv1
mode: enforce
mx: smtp.google.com
mx: *.aspmx.l.google.com
max_age: 604800

Practical verdict: relevant if you operate your own mail servers and want to ensure incoming email arrives encrypted over TLS, configured alongside the _mta-sts DNS TXT record.

pgp-key.txt, pgp.asc, and publickey.txt#

Hosting public PGP/GPG keys in plain text makes it easy for researchers to encrypt messages sent to the addresses in security.txt or verify software releases.

Production references:

The format follows standard OpenPGP ASCII Armor (-----BEGIN PGP PUBLIC KEY BLOCK-----).

ssh-key.txt and .keys endpoints#

While not a formal RFC, serving public OpenSSH keys (id_ed25519.pub) over plain text is a convenient developer habit.

GitHub and GitLab offer this natively by adding .keys to any profile URL:

When standing up a fresh VPS, a single terminal command imports your authorized keys:

curl -s https://github.com/torvalds.keys >> ~/.ssh/authorized_keys

jwks.json: public keys for JWT tokens#

Defined under RFC 7517, /.well-known/jwks.json distributes public cryptographic keys used by an identity provider to sign JSON Web Tokens.

Live examples:

Practical verdict: essential if you build OAuth2/OIDC authentication or decoupled APIs. Consuming services simply refresh their cache of public keys when keys rotate.

posh: delegated certificate verification#

POSH (RFC 7711) sits at /.well-known/posh/ and handles delegated TLS certificate validation when applications cannot access DNS TLSA records. Rarely encountered outside federated enterprise networks.

4) Advertising and AdTech: the truth about ads.txt, app-ads, and sellers.json#

This is where website owners waste the most time due to generic auditor warnings.

ads.txt: useful for programmatic auctions, useless for affiliates#

The ads.txt (Authorized Digital Sellers) standard was created by the IAB Tech Lab to combat inventory fraud in programmatic ad auctions. It publicizes which exchange accounts have legal permission to sell ad space on your domain.

Live examples:

The file uses comma-separated lines:

google.com, pub-1234567890123456, DIRECT, f08c47fec0942fa0
appnexus.com, 1234, RESELLER, f5ab79cb980f11d1

The practical truth:

app-ads.txt: mobile app equivalent#

The mobile sibling of ads.txt applies to apps on Google Play and Apple App Store. Because mobile apps cannot host web servers, ad buyers inspect the developer website listed in the app store to read /app-ads.txt.

Live examples:

Unless you publish mobile store apps monetized with in-app banner networks, you do not need this file.

sellers.json and buyers.json: the exchange side#

While ads.txt lives on publisher websites, sellers.json and buyers.json live on ad network domains:

They allow buyers to verify who receives payment and trace malvertising sources. Standard website owners never publish these files.

5) Identity and the fediverse: atproto-did, did.json, webfinger, and humans.txt bit rot#

Plain text records also serve as identity anchors across decentralized networks.

atproto-did and did.json: custom domains on Bluesky#

The AT Protocol (which powers Bluesky) lets you use your personal domain as your network handle (e.g. @perciocastelo.com.br).

The protocol defines an HTTP route at /.well-known/atproto-did containing a single line with your DID (did:plc:abcdef...). In practice, however, almost everyone (from individual developers to major news outlets) uses a DNS TXT record at _atproto.<domain>.

The reason is practical: the resolver queries DNS first. If DNS succeeds, the HTTP request never fires. The W3C did:web standard, on the other hand, relies on /.well-known/did.json to publish public keys:

keybase.txt: cryptographic proof of ownership#

Keybase pioneered using static plain text files to verify domain control. At /.well-known/keybase.txt or root, the owner hosts a PGP-signed statement linking their username to the site.

You can inspect the live proof on the Keybase creator's site: Chris Coyne keybase.txt.

webfinger: the phonebook of ActivityPub#

Standardized under RFC 7033, WebFinger resolves account handles (acct:user@domain) into profile endpoints, inbox paths, and public keys across the Fediverse (Mastodon, Lemmy, Threads).

Test it via curl against the Mastodon founder's instance:

host-meta and nodeinfo: federated host metadata#

The /.well-known/host-meta endpoint (RFC 6415) directs clients to discovery templates, while /.well-known/nodeinfo shares instance statistics (software version, active user counts, and protocols):

humans.txt and the rule of survival at web root#

The /humans.txt file was created by humanstxt.org in 2011 as a friendly gesture: while robots.txt talks to machines, humans.txt credited the designers, developers, and tools behind the website.

Over the years, the file quietly vanished. Major tech portals that once maintained it (including MDN Web Docs and A List Apart) dropped the file during static site generator and CDN migrations. Even the original initiative site (humanstxt.org) suffered bit rot.

This reveals a reality of modern web infrastructure: machine-critical files (ads.txt, security.txt, jwks.json) are fiercely maintained because downtime breaks revenue or communication. Files kept purely out of sentiment tend to disappear during the next repository overhaul.

When you tap a web link on your phone and it opens directly inside the installed app instead of a browser tab, these two files make it happen.

The /.well-known/assetlinks.json file creates a cryptographic bond between your website and an Android app, preventing rogue apps from intercepting your brand URLs:

Production examples:

The file links the Android package name to the SHA-256 certificate fingerprint from Google Play:

[{
  "relation": [
    "delegate_permission/common.handle_all_urls",
    "delegate_permission/common.get_login_creds"
  ],
  "target": {
    "namespace": "android_app",
    "package_name": "com.instagram.android",
    "sha256_cert_fingerprints": [
      "0B:20:9B:6C:5F:8B:D8:0F:7B:6A:9C:12:4D:2E:3F:4A:5B:6C:7D:8E:9F:0A:1B:2C:3D:4E:5F:60:71:82:93"
    ]
  }
}]

On iOS and macOS, Apple uses /.well-known/apple-app-site-association for Universal Links, Handoff, and shared credentials:

Production examples:

Apple enforces strict hosting rules: no file extension, Content-Type: application/json, direct HTTP 200 responses (no 301/302 redirects), and trusted public CA certificates.

Both files are only needed if you publish native mobile store apps.

7) Domain verification and payments: acme, search consoles, and Apple Pay#

Infrastructure providers frequently require small plain text files to verify ownership before enabling services.

ACME HTTP-01 challenge#

When issuing free SSL/TLS certificates via Let's Encrypt or Certbot, the automated agent places a temporary plain text token in /.well-known/acme-challenge/<token>. The certificate authority validates this endpoint over HTTP before signing the certificate. This process runs automatically.

Search engine verification files#

To prove ownership without editing DNS records, search engine tools offer static files:

Practical verdict: once ownership is verified, most teams migrate to DNS TXT records and remove the static file to keep repositories clean.

apple-developer-merchantid-domain-association#

Enabling Apple Pay in Safari requires hosting a PKCS #7 signed binary file provided by Apple at /.well-known/apple-developer-merchantid-domain-association. Without it, Safari will not render the Apple Pay button:

8) Service discovery and network policies: caldav, traffic-advice, and telemetry#

Manifests under /.well-known/ also help network clients discover services or declare prefetch boundaries.

caldav and carddav: calendar and contact redirects#

Standardized under RFC 6764, /.well-known/caldav does not serve static content. It operates as an HTTP 301 or 307 redirect pointing to your organization's WebDAV calendar endpoint.

Calendar apps on iPhone, Mac, and Thunderbird query this path to find sync URLs automatically:

traffic-advice: Chrome prefetch control#

Google Chrome attempts to prefetch search results so clicks feel instantaneous. The /.well-known/traffic-advice file lets website owners accept or throttle speculative prefetching.

An interesting quirk: testing google.com/.well-known/traffic-advice returns 404 because Google Search is the requester, not the destination. The specification dictates that a 404 on the target site means prefetching is allowed by default.

network-error-logging and trust-tokens#

9) Privacy controls and browser shortcuts: gpc.json, passwords, and cache revalidation#

Standards designed to respect user privacy and streamline browser workflows.

gpc.json: Global Privacy Control#

The /.well-known/gpc.json file declares that a website respects Global Privacy Control (GPC) signals sent by browsers like Brave and Firefox, complying with privacy mandates like CCPA and GDPR:

The structure is straightforward:

{
  "gpc": true,
  "version": 1,
  "last-update": "2024-01-15",
  "privacy-policy": "https://www.exemplo.com/privacy-policy"
}

The older dnt-policy.txt (Do Not Track) was discontinued by W3C in 2019 and superseded by GPC.

change-password: the password manager shortcut#

Under RFC 8615, /.well-known/change-password is an HTTP 302 temporary redirect pointing straight to the account password change screen.

When Chrome, Safari, 1Password, or Bitwarden detect a leaked password, they present a one-click button targeting this URL:

If your website has user accounts, adding this redirect in Nginx or Apache takes thirty seconds and noticeably improves security UX.

reload, revalidate, and pay#

10) Operations and deployment pipelines: health.txt, version.txt, and licenses#

Simple plain text files make server maintenance and deployment audits remarkably easy without extra dependencies.

health.txt and status.txt: lightweight probes#

Hitting a heavy application route that queries databases every ten seconds wastes resources.

A simple /health.txt file returning OK lets external monitors (like UptimeRobot) check web server availability with minimal overhead:

location = /health.txt {
    access_log off;
    default_type text/plain;
    return 200 "OK\n";
}

The /status.txt file follows a similar pattern to expose basic uptime or server connection metrics.

version.txt and build.txt: deployment stamps#

During automated builds, writing a static /version.txt file with the Git commit hash and build timestamp makes production tracking trivial:

version=2.18.4
commit_sha=a3f92b71c82d90ef01b88e458e0a7df8921a9c4b
build_time=2026-09-25T23:58:12Z
branch=main

A quick curl immediately confirms whether a deployment reached all nodes. The build.txt file serves a similar role, documenting compiler versions and build environments for supply-chain security.

LICENSE.txt, CHANGELOG.txt, and AUTHORS.txt#

The classic open-source triad. Web bundlers frequently package a LICENSE.txt or 3rdpartylicenses.txt at the web root to comply with legal redistribution clauses of third-party JavaScript libraries.

The practical filter: what I keep on my domain and what you can ignore#

After testing all these standards against real server traffic, the conclusion is clear: most automated audit warnings are pure noise.

On my own domain, I keep things lean:

  1. robots.txt: Essential to guide crawlers and declare the sitemap.
  2. security.txt: Invaluable under /.well-known/ to provide a responsible reporting channel for security researchers.
  3. llms.txt: A clean Markdown addition at the web root for AI search tools.
  4. health.txt and version.txt: Lightweight operational files to monitor uptime and check deployments via terminal.
  5. atproto-did: Only if you want your domain handle on Bluesky (though DNS TXT is even simpler).

Everything else depends entirely on specific infrastructure you run:

Keeping your web root uncluttered, protected by an edge layer like Cloudflare, and publishing only what your setup actually needs saves time and keeps you from chasing empty compliance alerts.

Ready-to-use prompt to audit your project with AI and export a PDF report#

To save you from manually inspecting each file and RFC guideline across every website or codebase you manage, I assembled a battle-tested technical prompt ready for immediate use.

The workflow is straightforward: copy the instructions block below and paste it into your preferred AI assistant (such as Claude, ChatGPT, Gemini, or DeepSeek). The AI will act as a senior web architect and security auditor, ingest the guidelines covered in this guide, interview you about your technical stack (or inspect the URL and repository files you attach), filter out unnecessary file bloat, generate production-ready file contents tailored to your domain, and produce a structured technical audit report with a script to export everything directly to PDF.

Act as a Senior Web Architect, Application Security Specialist, and Web Standards Engineer (RFCs and IANA specifications).

Your goal is to audit my project/web domain and determine with surgical precision which plain text files (.txt), manifests in the /.well-known/ directory, and web metadata standards must be implemented, which are recommended best practices, and which MUST BE STRICTLY IGNORED to prevent unnecessary server bloat and false compliance alerts.

At the conclusion of the audit, you will provide the complete, ready-to-deploy content for every applicable file, along with a self-contained executable script (Python with ReportLab or WeasyPrint) OR a standalone HTML5/CSS3 printable document with (@media print) styles to generate a professional executive PDF report containing all findings and snippets.

---

### STEP 1: PROJECT INTAKE

If I have not already provided these details in my initial message, pause and ask me:
1. What is the primary domain name of the project (e.g., example.com)?
2. What is the nature of the project? (Content blog/media portal, SaaS/web application, REST/GraphQL API, e-commerce store, static landing page, open-source repository)?
3. What is the technology stack? (WordPress, Next.js, Laravel, Django, Node.js, static HTML, etc.)?
4. Does the site monetize with programmatic ad networks (Google AdSense, AdX, OpenX)?
5. Does the project feature a companion native mobile app (Android/iOS) requiring Deep Linking / Universal Links?
6. Does the domain host its own mail transfer agent (Postfix, Exim) or rely on third-party cloud mail (Google Workspace, Proton, Fastmail)?
7. Does the domain act as an identity provider (OAuth 2.0 / OIDC) or merely consume external providers?
8. Do you have repository files, directory trees, or URLs you would like to provide right now for direct review?

(If I already supplied these details or the target domain URL upfront, skip the questions and proceed directly to the audit).

---

### STEP 2: TECHNICAL AUDIT MATRIX

Evaluate each of the following files and standards against the specific architecture of my project:

1. Search Engine Discovery & Indexing:
   - robots.txt (User-agent rules, Disallow, Allow, and Sitemap directive)
   - sitemap.xml and plain text sitemap.txt
   - urllist.txt (Classify as legacy/deprecated or explain equivalence)

2. Artificial Intelligence & Large Language Models:
   - llms.txt (Concise Markdown index at root or /docs/)
   - llms-full.txt (Complete context document for LLMs)
   - ai.txt (Classify as non-standard myth/unratified proposal)

3. Programmatic Advertising & AdTech:
   - ads.txt and app-ads.txt (IAB Tech Lab authorized sellers)
   - sellers.json and buyers.json (Ad exchanges and intermediaries only)

4. Security, Contact & RFC Standards:
   - /.well-known/security.txt (RFC 9116: Contact, mandatory Expires date, Canonical, Preferred-Languages, Policy, Encryption)
   - /.well-known/security.txt.sig (Detached PGP signature)
   - /.well-known/mta-sts.txt (RFC 8461 for self-hosted mail servers)
   - humans.txt (Team credits)

5. Public Cryptographic Keys:
   - pgp-key.txt, pgp.asc, or /.well-known/openpgpkey/
   - publickey.txt or authorized SSH keys

6. Identity, Federation & Social Protocols:
   - /.well-known/jwks.json and openid-configuration (JSON Web Key Set and OIDC)
   - /.well-known/webfinger (RFC 7033 for Fediverse / Mastodon)
   - /.well-known/atproto-did (Domain verification for Bluesky / AT Protocol)

7. Mobile App Integration & Deep Linking:
   - /.well-known/assetlinks.json (Android App Links)
   - /.well-known/apple-app-site-association (iOS Universal Links)

8. Operations, CI/CD & Observability:
   - health.txt or status.txt (Ultra-lightweight ping probes)
   - version.txt or build.txt (Git commit hash and deployment timestamps)
   - LICENSE.txt and 3rdpartylicenses.txt

---

### STEP 3: DIAGNOSTIC AUDIT & GENERATED ARTIFACTS

Organize all audit findings into three unambiguous tiers:
- CRITICAL / MANDATORY: Essential files strictly required for this project type.
- RECOMMENDED: Best-practice files that improve security, AI visibility, or observability without operational maintenance debt.
- UNNECESSARY / NOT APPLICABLE: Files that must NOT be created, including an explicit technical rationale explaining why they would create pointless noise.

For every file marked Critical or Recommended:
1. Provide the exact file path where it belongs (e.g., /public_html/.well-known/security.txt).
2. Provide the complete, production-ready, validated code populated with my domain's real information.
3. If applicable, provide the web server block (Nginx, Apache, or Cloudflare Transform Rules) to serve it with Content-Type: text/plain; charset=utf-8.

---

### STEP 4: EXECUTIVE PDF REPORT GENERATOR

To enable formal documentation for clients, stakeholders, or infrastructure audits, deliver at the end:
1. The full executive audit report.
2. A self-contained Python script (using reportlab or weasyprint) OR a standalone HTML5/CSS3 template with @media print rules ready to open in any web browser and save via Ctrl + P -> "Save as PDF".
The PDF report must feature:
- Executive header with domain name, audit timestamp, and report version.
- Summary table with file status badges (Mandatory, Recommended, Skipped).
- Full syntax-highlighted code blocks for each generated file.
- Actionable implementation checklist and server header configuration.

Was this article helpful?

Leave a quick reaction to help prioritize future technical guides:

See also

Infrastructure Why Split Sitemaps: Index Architecture for Technical SEO
Infrastructure Anatomy of a SIGKILL in login failure daemon: debugging LFD service startup failure on Linux
Infrastructure The sshd_config illusion: how to actually ensure SSH authentication security on cPanel and CloudLinux
CC BY-NC

This post is licensed under CC BY-NC.

Comments

Join the discussion below.

By commenting, you agree to our Terms of Service and Privacy Policy.

0 comments