Link not located in the respons in Screen Scraper
Hi
I have a problem. When i accesses a webpage from IE there is a link around a header. When i accesses it from Screen Scraper the page does not contain the link. There is a lot of different .js files that are loaded after the main page. How can i get the link to be accesiable in SS?
Hans
SS won't try to do anything
SS won't try to do anything with Javascript, which you were keen to observe-- pages like this make the task unfortunately complicated, but if you can simulate what the relevant javascript does, using SS scripts, you can assemble the link yourself, in most cases.
You might find that you won't have the luxury of extracting an entire link to be plopped into a scrapeableFile. If you can proxy the act of clicking on the link in your browser, and you use the proxy'd URL as your template, and try to sift through the javascript to find where it gets values that make up that URL.
Good luck :)
Tim
Simple Java
Hi Tim
I tried to access the page from plain java and wrote a little test, to see how java handles the page. This is what I tried:
URL u = new URL("http://www.xx.zz/yyyy.html");
HttpURLConnection con = (HttpURLConnection) u.openConnection();
InputStream stream = con.getInputStream();
InputStreamReader reader = new InputStreamReader(stream);
BufferedReader bureader = new BufferedReader(reader);
String l;
while((l = bureader.readLine()) != null){
System.out.println(l);
}
The output is with the right links and everything. Is there some way I can accomplish this in the scraper? I could try and spend some time writing a proxy server that handles this, but thought there might be an easier solution.
Hans
Still problems
Hi Tim
I tried to have a closer look at the javascripts, but still have problems finding a way to do it.
Can you please give some more hints on how to accomplish the task?
Hans