TBF, pushing a site to the public while adding a “no scrapping” rule is a bit of a shitty practice; and pushing it and adding a “no scrapping, unless you are Google” is a giant shitty practice.
Rules for politely scrapping the site are fine. But then, there will be always people that disobey those, so you must also actively enforce those rules too. So I’m not sure robots.txt is really useful at all.
How would a site make itself acessible to the internet in general while also not allowing itself to be scraped using technology?
robots.txt does rely on being respected, just like no tresspassing signs. The lack of enforcement is the problem, and keeping robots.txt to track the permissions would make it effective again.
I am agreeing, just with a slightky different take.
TBF, pushing a site to the public while adding a “no scrapping” rule is a bit of a shitty practice; and pushing it and adding a “no scrapping, unless you are Google” is a giant shitty practice.
Rules for politely scrapping the site are fine. But then, there will be always people that disobey those, so you must also actively enforce those rules too. So I’m not sure robots.txt is really useful at all.
How would a site make itself acessible to the internet in general while also not allowing itself to be scraped using technology?
robots.txt does rely on being respected, just like no tresspassing signs. The lack of enforcement is the problem, and keeping robots.txt to track the permissions would make it effective again.
I am agreeing, just with a slightky different take.