Variable Proxy Server?
Is it possible to use a session variable to populate the Proxy Port and Host setting in a scrape? By using something like ~#PROXYHOST#~ and ~#PROXYPORT#~?
I've had luck passing username and password into the authentication settings for a scrapeable file like that, but this is at the session level.
Short of that, is there a way I can set a session's proxy by script?
Variable Proxy Server?
Robert,
Have a look at the newest up-and-coming feature that is still in development.
Anonymization:
http://dev.screen-scraper.com/support/docs/anonymization.php
You may find something useful here even if main purpose doesn't seem to fit your needs.
-Scott
Variable Proxy Server?
Robert,
Likely, no since screen-scraper would be reading in that file at the time that the application loads. Much sooner than any scraping sessions are read in.
You can play around with it but you may need to create multiple instances of screen-scraper on your one box and spawn them with their own proxy settings to accomplish this. Same approach would be necessary if you were dealing with multiple character encodings.
-Scott
Variable Proxy Server?
That's awesome!
Is there any way to target a specific scrape to a specific proxy by means of a session variable? Or can I set the proxy via session.setProxy() or something like that at the beginning of a scrape?
Variable Proxy Server?
Robert,
One feature of screen-scraper that I was previously not aware of is the ability for you to create a text file, name it proxy.txt and place it in your screen-scraper installation folder.
Within the file you can list as many proxies as you would like. When screen-scraper detects the existence of the file it will read it in and iterate through the values until one works.
The format may look something like this:
http://localhost:1259
http://www.screen-scraper.com:6349
http://www.scrapbookfinds.com:2349
This will over-ride the proxy settings you may have set in the main settings dialog.
Give it a try and let us know if it works to solve your problem.
-Scott