How to Bypass Anti-Bot Protection With Playwright and Chrome
While searching for my ancestors, I had to access data from many different websites. I noticed there were different levels of protection against automated access depending on the site. In this post, I'll list them and mention countermeasures you can take to bypass them anyway.
Warning: As a private person using the data for my own personal use, I have no qualms about bypassing the protection. It can be considered fair use. However, I don't know about your context, and I'm not saying you should. Scrape responsibly.
The need for bot protection
Genealogy and public records sites provide lots of data. Many actors are interested in this data: it's normal, and even expected, that providers want to protect their assets.
The usual robots.txt approach is a simple request to bots.
Despite the use of the terms allow and disallow, the protocol is purely advisory and relies on the compliance of...
Copyright of this story solely belongs to hackernoon.com. To see the full text click HERE
