Skip to main content

§ Machine index

Tutorials
83
Categories
10
Blog posts
31

AI Content Index

This page indexes all content on LearnWebCraft for AI crawlers, search engines, and content discovery.

§01Categories

Categories

§04Crawlers

Allowed AI crawlers

LearnWebCraft welcomes AI crawlers configured in robots.txt:

OpenAI

  • GPTBot
  • ChatGPT-User
  • OAI-SearchBot

Anthropic

  • ClaudeBot
  • Claude-Web
  • anthropic-ai

Google

  • Googlebot
  • Google-Extended

Other

  • PerplexityBot
  • YouBot
  • cohere-ai

Last updated: 2026-11-18T11:52:04.647Z · 114 total articles