就現況而言，我對 LLMs(大型語言模型)抱持著相當矛盾的看法。這篇札記不會深入探討我對此的想法——我也不想展開那場討論。

不過，雖然我在這裡做的並非什麼尖端研究，但我確實用心為人類而發表內容。我喜歡知道，自己在這裡花 10 分鐘寫下的東西，未來或許能為幾個人節省時間。

我這麼做，不是為了讓別人拿去建立模型，然後再以訂閱制拿去販售。

我新的 robots.txt 現在已排除多個已知 LLM 爬蟲的 user agent。列出的大多數爬蟲都已獲得確認，且每一項我都附上了來源。新聞媒體的 robots.txt 也相當好用，因為他們同樣會封鎖 LLMs。

如果你也想這麼做，歡迎直接拿我的去用！

# I know this can just be _totally_ ignored by crawlers
# But let's hope they behave well :)

# Code: https://github.com/ellie/notes
# Source: https://darkvisitors.com/

# OpenAI, ChatGPT
# https://platform.openai.com/docs/gptbot
User-agent: GPTBot
Disallow: /

# Google AI (Bard, etc)
# https://developers.google.com/search/docs/crawling-indexing/overview-google-crawlers
User-agent: Google-Extended
Disallow: /

# Block common crawl
# I have mixed feelings on this one, but many models are trained on this data
# It is also used to bootstrap new search indices though
# https://commoncrawl.org/ccbot
User-agent: CCBot
Disallow: /

# Facebook
# https://developers.facebook.com/docs/sharing/bot/
User-agent: FacebookBot
Disallow: /

# Cohere.ai
# https://darkvisitors.com/agents/cohere-ai
User-agent: cohere-ai
Disallow: /

# Perplexity
# https://docs.perplexity.ai/docs/perplexitybot
User-agent: PerplexityBot
Disallow: /

# Anthropic
# https://darkvisitors.com/agents/anthropic-ai
User-agent: anthropic-ai
Disallow: /

# ...also anthropic
# https://darkvisitors.com/agents/claudebot
User-agent: ClaudeBot
Disallow: /