minus-squareMichaelMuse@programming.devtoSelfhosted@lemmy.world•Anubis is awesome! Stopping (AI)crawlbotslinkfedilinkEnglisharrow-up1·2 days agoI think AI can provide an interface to let user submit the site for crawling, such as some website scanner doing, like urlscan. Otherwise the site can reject the AI crawler. linkfedilink
I think AI can provide an interface to let user submit the site for crawling, such as some website scanner doing, like urlscan. Otherwise the site can reject the AI crawler.