Home » Forums » screen-scraper » screen-scraper support for licensed users

Ability to change URL dynamically

Is it possible to modify the current url dynamically? I see that I can get the value of the current url, but I don't see a way to set it or modify it. I need to add a token at runtime in some cases before a scrape is run. The token is part of the url, not a parameter. Is there anything that I can do?

Thanks.

UPDATE: Ah. I see that I can only get the URL after the file has been scraped. Is there a way to intercept and modify the current URL *before* scraping?

‹ Complicated Site with Frames Looks Good in Proxy Session but not in Scraping Session OutOfMemory Error when scraping page with many results ›
robind on 11/28/2011 at 5:13 pm

There's no method other than

There's no method other than editing the URL string with Java tools, but if you could share a sample URL I might be more help. There are some that have an ID in a URL that you can just omit and it will still work. Other times one can compose a URL more easily.

 jason on 11/29/2011 at 12:46 pm

This occurs when we need to

This occurs when we need to connect to a site via a proxy. We use a proxy tool created with Apache modules to do any proxying necessary when connecting to the various sites that we need to scrape. When connecting to a site via this proxy, we add a token to the url that we read from configuration files at runtime. I would need to add this token to the url that is stored in the screen-scraper scraping session. Is there a way to do that?

 robind on 11/29/2011 at 1:19 pm

Maybe a little more info will

Maybe a little more info will help you to help me. :)

I have a handful of scraping sessions set up. I am invoking them via an external java app. Some sites require a proxy and for those I have to add a token to the url at runtime. I set the host and the port using session.setExternalProxyHost/Port and that is working just fine. I can manually add the token to the URL that's set for the scraping session, but that's not ideal. Please let me know if what I need to do is still not clear!

 robind on 11/29/2011 at 6:33 pm

robind, Try passing your

robind,

Try passing your dynamic variable to your URL in the URL field under the General tab of the scrapeable file. For example.

Script to run before scrapeable file:

session.setv("foo","bar");

Scrapeable File:

URL: http://www.mydomain.com/~#foo#~/abc.html

Does that help?

-Scott

 swilsonmc on 11/29/2011 at 9:17 pm

I'll give it a try. I didn't

I'll give it a try. I didn't realize I could put a variable in the URL Field for the scrapeable file. Thanks for the info!

 robind on 11/30/2011 at 10:29 am