Scraping generated source trouble
Hi,
I am trying to scrape the Bid history table from www.swoopo.com (if you click on any random auction, it's the table to the mid right, with BID, BIDDER, TYPE). I was expecting this to be pretty easy as it's obviously just text, but I can't for the life of me find it in the source. I think it must be dynamically generated by the javascript, but I would have thought it would end up in the source code. Anyone any idea how I would get at this data?
Cheers
javascript request
Phaedos,
So, from what I've seen of the site, it looks like there is a http request that goes out every time a bid is placed. Either that or it's on a timer and the http request brings back the bid history every 5 seconds or something.
I'd proxy this page and pay careful attention to the requests that get sent out (there could be a lot of them considering the heavy nature of this site). Find the one that has to do with the updating of the bid history and you are going to mimic this request every 5 seconds or so in screen-scraper. This will begin to return the bid history to your scrapeable file and you can then scrape it. The information could be sent back in a variety of formats, but they should all be manageable.
That's my $.02
Thanks
scraper