Scraping javascript results
I'm looking to log the football results on http://www.livescore.com/. If I display the page in a browser and use "View Source", I can see the required results without problems. However, when running the same page through screen-scraper's proxy, I get a body consisting og just question marks.
Scraping javascript results
rosman71,
Ah, I should have caught that. Try this...
********************************
But, be careful when running a loop this large. screen-scraper can't always stop the process once it's been dispatched to the JVM.
I recommend testing with a shorter range, say:
9550000000000 - 9550000000010
********************************
long stop = Long.parseLong("9559999999999");
for (i=start; i<=stop; i++)
{
session.log("***Current number is " + i);
// Optionally set the current number to a variable
session.setVariable("myNumber", i);
// Optionally call a scrapeable file for each number
session.scrapeFile("My Scrapeable File");
}
-Scott
Scraping javascript results
thanks.
BTW, the parameter required a string instead of integer.
How do I use the input as string with incremental as per your code below
for (int i=9550000000000; i<=9559999999999; i++)
{
session.log("***Current number is " + i);
// Optionally set the current number to a variable
session.setVariable("myNumber", i);
// Optionally call a scrapeable file for each number
session.scrapeFile("My Scrapeable File");
}
Scraping javascript results
rosman71,
Something like this should work.
{
session.log("***Current number is " + i);
// Optionally set the current number to a variable
session.setVariable("myNumber", i);
// Optionally call a scrapeable file for each number
session.scrapeFile("My Scrapeable File");
}
input with a range of valules
Hi,
I'm trying to use the shopping cart. But I'm still not clear how to provide an input range to it. How would I able to do it?
I was trying to collect information from http://www.halal.gov.my/directory/barcode.php
using the input range from 9550000000000 to 9559999999999
Scraping javascript results
You're quite welcome. :)
Scraping javascript results
Scott, your suggestion works perfectly. I can now scrape the SID automatically to retrieve the page in the next scrapeable file. Thank you very much.
Scraping javascript results
Just made a quick test, and your session seems to work :-) I'll dig more into it during the weekend.
Scraping javascript results
manscher,
Well, they get an A for trying (for now). This site is attempting to detect software like screen-scraper by requiring the client (browser) to submit a form via JavaScript and pass a cookie. The only thing screen-scraper couldn't do without a little help was submit the form.
This is because the form submission was invoked via JavaScript in an onLoad call from the body tag. All you need to do is mimic this action by creating an additional scrapeable file that contains the one post parameter that is part of the form submission and have the URL for the scrapeable file be the URL in the form action (which contains a GET parameter).
You can download my sample scraping session [url=http://www.screen-scraper.com/xfer/livescore.com.zip]here[/url].
It's likely this site will change their tactics to prevent screen scraping. So, be aware that you may be hunting a moving target.
If you're still only seeing ???'s in screen-scraper, you'll need to change the default character encoding to something like ISO-8859-1.
-Scott