Home » Forums » screen-scraper » screen-scraper public support

Evil AJAX

Hi there.

I've been scraping away merrily and everything has been working fine, until I wanted to scrape the brand names from this site. The problem is the site offers users to view the products either from TYPE (the default) or from Brand (A - Z)

http://www.mysupermarket.co.uk/shelves/Condiments_in_Tesco.html

Problem is when you select Brand (A - Z) from the dropdown, no URL changes. I've worked out that it's done using an AJAX call like this:

http://www.mysupermarket.co.uk/Ajax/SaveGroupBy.aspx?GroupBy=2366

This then sets a cookie or something in the session variable.

Where can I put this bit of information so that screen-scraper picks this up and processes it?

Many thanks

‹ Scraping all URL's on a site Regarding Downnload images from the site ›
mrpugster on 03/28/2011 at 9:26 am

mrpugster, You're correct.

mrpugster,

You're correct. If you make the appropriate request to SaveGroupBy.aspx then the server will set a cookie (which screen-scraper handles automatically). Then, when you request the results page (Condiments_in_Tesco.html) now with the cookie set it will display the results sorted A-Z.

When requesting the SaveGroupBy.aspx there are actually four GET paramters (GroupBy, Shelf, ts, PageEncodedUrl). I found that the first two and last parameter can be scraped from the original results page (Condiments_in_Tesco.html), stored in a session variable and passed to SaveGroupBy.aspx. You can leave the value of ts untouched.

After making your request to SaveGroupBy.aspx simply request Condiments_in_Tesco.html and you should see the contents sorted appropriately.

-Scott

 swilsonmc on 03/28/2011 at 11:15 am

Thanks Scott, I'm a little

Thanks Scott, I'm a little lost on how to do this. My orginal scrape didn't need to set any parameters and I didn't really have to use the proxy so I pretty much skipped those parts. I think I will need to revisit the tutorial.

Do you recommend any part?

 mrpugster on 03/28/2011 at 12:13 pm

Ah, yes. The tutorials are

Ah, yes. The tutorials are very important. You'll definitely want to proxy the site. To learn how that works just start at the first tutorial and stop when you feel that you have what you need.

http://community.screen-scraper.com/Tutorials_Menu

-Scott

 swilsonmc on 03/28/2011 at 4:07 pm