Proxy connections are getting blocked from web sites
I've noticed that we are getting blocked on several sites consistently since upgrading to the past few versions of screen scraper. Zillow, Trulia and a few others seem to be able to detect the IP address even though we have proxies in use.
Search results: Resolved URL: http://www.trulia.com/validate.php?tst=h&display=for%20sale&search=01420&type=&min_price=min&max_price=max&min_num_beds=0&min_num_baths=0&min_size=0
Search results: The request timed out while requesting the URL: http://www.trulia.com/validate.php
Search results: There was an error on the last request. Now making attempt number 1 of 20 requests.
Search results: Resolved URL: http://www.trulia.com/validate.php?tst=h&display=for%20sale&search=01420&type=&min_price=min&max_price=max&min_num_beds=0&min_num_baths=0&min_size=0
Search results: The request timed out while requesting the URL: http://www.trulia.com/validate.php
Search results: There was an error on the last request. Now making attempt number 2 of 20 requests.
I've seen an entire run get blocked when proxies are on.
Turn off the proxy and the scraper works.
Use a service like HMA VPN and it may or may not work again.
Any ideas on what I could try to get around the IP / Proxy usage blocking?
I wish I knew a way. I know
I wish I knew a way. I know someone who has been scraping Zillow for a few years, and the IP blocking is a major, ongoing bother.
All I can tell you is to try to look more human. Add pauses, change up user agents with the IPs, and don't request anything that you don't really need.