minus-squaregaylord_fartmaster@lemmy.worldtoTechnology@lemmy.world•Cloudflare plans marketplace to sell permission to scrape websiteslinkfedilinkEnglisharrow-up50·6 days agoThey’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent. linkfedilink
They’re already ignoring robots.txt, so I’m not sure why anyone would think they won’t just ignore this too. All they have to do is get a new IP and change their useragent.