robots.txt blocks CCBot (Common Crawl - feeds many open-source LLMs)

CCBot is the crawler behind Common Crawl, a large, free, publicly available archive of web pages that many organisations use to train large language models (LLMs), the AI systems behind tools like open-source chatbots. This check found robots.txt disallowing CCBot, so none of your pages can enter that archive. Common Crawl feeds the training data behind many open-source AI models, including well-known ones like Llama and Mistral, as well as smaller research projects. Blocking CCBot is low-severity individually, since it affects long-tail AI distribution rather than any single commercial product you'd recognise by name, but it's a one-line fix with essentially no downside if AI visibility is a goal. Your developer needs about 5 minutes. Add an allow rule for CCBot in robots.txt, unless you're deliberately keeping your content out of open-source AI training data. This is a low-priority, low-risk fix: there's no realistic downside to allowing a well-documented, non-commercial research crawler.