对于 LLM(大语言模型)的现状，我的看法非常复杂。这篇笔记不会深入探讨我在这个问题上的想法——我不想展开那种讨论。

不过，虽然我这里做的算不上什么前沿研究，但我确实是用心为人类读者发布内容的。我很高兴知道，我花在这里写作的 10 分钟，将来也许能为一些人节省时间。

我做这些不是为了让人拿去训练一个模型，然后再以订阅的形式卖出去。

我新的 robots.txt 现在已排除了一批已知 LLM 爬虫的 user agent。我所列出的爬虫大多已经过确认，并且我为每一个都附上了来源。各大新闻媒体的 robots.txt 也相当有用，因为它们同样屏蔽了 LLM。

如果你也想这样做，欢迎直接拿去用！

# I know this can just be _totally_ ignored by crawlers
# But let's hope they behave well :)

# Code: https://github.com/ellie/notes
# Source: https://darkvisitors.com/

# OpenAI, ChatGPT
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /

# Google AI (Bard, etc)
# https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
Disallow: /

# Block common crawl
# I have mixed feelings on this one, but many models are trained on this data
# It is also used to bootstrap new search indices though
# https://commoncrawl.org/ccbot
User-agent: CCBot
Disallow: /

# Facebook
# https://developers.facebook.com/docs/sharing/bot/
User-agent: FacebookBot
Disallow: /

# Cohere.ai
# https://darkvisitors.com/agents/cohere-ai
User-agent: cohere-ai
Disallow: /

# Perplexity
# https://docs.perplexity.ai/docs/perplexitybot
User-agent: PerplexityBot
Disallow: /

# Anthropic
# https://darkvisitors.com/agents/anthropic-ai
User-agent: anthropic-ai
Disallow: /

# ...also anthropic
# https://darkvisitors.com/agents/claudebot
User-agent: ClaudeBot
Disallow: /