Ability of screen-scraper to simulate user interaction with a web site
I have a basic question about the ability of screen-scraper to simulate user interaction with a web site:
Can you pass parameters via the HTTP GET method to ALL web sites in order to insert data into fields? Or does a web site have to be coded to accept these parameters? Are there any web sites that screen-scraper won't work with?
Is there anything that simulates clicking a specific button on a web page?
Buttons and forms are all
Buttons and forms are all just HTTP requests on the back end, so screen-scraper doesn't actually click anything, it just submits the HTTP request that the button would.
We've had a few websites that we couldn't scrape, and even most of those weren't impossible but rather prohibitively difficult, but they are very few and far between. Most websites can be scraped.
Here are some notes of things that would make it hard: http://blog.screen-scraper.com/2009/08/17/further-thoughts-on-hindering-screen-scraping/