Posting data to a web form?
Ok, I got the scraper session that I needed working great... the problem is, it's finding so many spammer domains that I'm having problems keeping up when manually submitting them to SiteAdvisor.com!
So, I wanted to set it up so that I just drop the list of spammer domains into a .txt file, fire up Screen-Scraper, and let it submit them for me (or, more elegantly, have it read the .txt file that Screen-Scraper is saving the discovered spammer domains into, and remove each domain from the list as it submits comments for them).
Here are the caveats
1) It has to be able to log in with my user name and password to the SiteAdvisor.com website.
2) It has to handle the cookies from SiteAdvisor.com (I think, from reading the man pages that Screen-Scraper handles cookies automatically... correct me if I'm wrong).
3) It has to be able to check the page for the spammer domain in question to determine if I've already submitted a comment for that domain, and if so, to not submit another one.
4) It has to be able to save the domains it's submitted comments for to another .txt file, so I can cut-and-paste them into the large list of spammer domains I keep for the spammer I'm currently pursuing (or, more elegantly, to save those submitted domains directly into the large list, in alphabetically arranged order).
I fired up my Microsoft Fiddler proxy and grabbed the data stream from a submission to SiteAdvisor.com, so I know exactly what needs to be submitted, I just don't know how to go about doing it programmatically from Screen-Scraper. I have some previous experience in posting data to forms using Javascript (I developed Refi Retaliator II), but I'm having trouble getting started on this project.
Anyone got any tips, hints, etc.?
Posting data to a web form?
Hi,
That all sounds perfectly doable, but would require at least a little bit of programming to write the data to the text files. For some sample code, you might check our third tutorial (here) as well as our seventh (here). Rather than using a different proxy, I'd recommend using screen-scraper's proxy to take the approach described in the third tutorial. In answer to your question on cookies, screen-scraper will handle cookies automatically as long as they're not set on the client-side via JavaScript. In those cases you'll need to use screen-scraper's session.setCookie method (here).
Not knowing how much programming background you have I'm uncertain what else to suggest. Feel free to post specific questions if you run into snags.
Kind regards,
Todd Wilson