dynamic proxy change
I know you can designate multiple proxies with a text file in the SS directory but I would imagine this file is read when SS starts and doesn't refresh while it's running. Is there a method (probably undocumented) for setting the external proxy? This would be really handy because then I could switch the proxy on the fly. There a couple of scenarios where you might not know the proxy IP until after a session has started, i.e. scanning for new proxies, spawning new ones yourself on EC2 etc...
If proxy is set per session and can't be set from within the session that's not necessarilly a problem as I could be using that session to spawn other sessions so could designate their proxy before starting them.
cheers
I had to do something like
I had to do something like this before, where I needed to spawn a variable number of scrapes, and a variable number of proxies to go with it. Once I received the signal that I had enough proxies, I had to ask what the host/port combinations were, so that I could dynamically set the external proxy settings on each scrape.
Currently, you can only change the setting in a script which runs "Before scraping session begins". If you change it at any other time, it won't stick.
To change the values on the fly, try the following:
session.setExternalProxyHost("host);
session.setExternalProxyPort("port");
In my case, I was determining what the host and port were in a controller session, which spawned dynamic scrapes, passing them session variables containing the information they would need, to set their hosts/ports by themselves.
Tim
That's perfect Tim. I'm
That's perfect Tim. I'm using pretty much the parent/child model you've described except I'm using a mysql db to pass parameters back and forth so I can get around the problem of lazyscrape sessions being inaccessible to the calling object... and it's useful because I can interfere with things by editing the db directly while it's in progress...