Require help/suggestions in scraping following sites

It sounds like you are trying to scrape http://www.bananarepublic.com/customerService/storeLocator.do for each zipcode in your file and that the problem is that you can't seem to create a scrapeable file that then scrapes all the detail information (under the "directions" links) for each store in the zipcode. The reason for this, I understand, is that the URLs for those links change based on some program internal to the server.

I looked at the URL under the "directions" link as represented in the raw HTML of the site and compared it with the link that my browser shows in the URL field and found that the raw HTML represented &'s with "&".

I found that if you scrape the URL under the "directions" link and then run it through a script that replaces all the "&" with "&" and then put it into the URL of the scrapeable file that tries to scrape the page resulting from the "directions" link, then it works.

I included the test scraping session that I made to scrape the stores in the 95132 zipcode. It's in raw XML form, so all you need to do it cut and paste the XML below into an editor, rename it to "br.xml", and import it into your screen-scraper workbench.

-Alan

BR1201http://206.231.92.100/StoreLocator/BRPrxInput.aspx?gapcountry=USA&brand=BRSearch
name="__VIEWSTATE" value="~@VIEWSTATE@~"VIEWSTATE

http://206.231.92.100/StoreLocator/BRPrxInput.aspx?gapcountry=USA&brand=BRSearch results
directionsGet linkDETAILS_LINK

controlAddress:txtAddressPOSTcontrolAddress:txtCityPOSTcontrolAddress:txtZipPOST95132buttonFind.xPOST34buttonFind.yPOST6controlAddress:listStatePOST__VIEWSTATEPOST~#VIEWSTATE#~localePOSTen-UScheckProperties:0POSTon

http://206.231.92.100/StoreLocator/BRPrxDriveInput.aspx~#DETAILS_LINK#~Details

Could you list the URLs in order that you're trying to scrape as well as any special parameters that each page needs? Also, it would help me to know the exact page you're trying to get data from and the data you're trying to get from that page.

Thanks,
Alan

Hi,

If you are trying to scrape http://www.bananarepublic.com/customerService/storeLocator.do using a list of zipcodes that you have, I would suggest the following:

1) Create a scrapeable file from the file which lists all the zipcodes and put an extractor pattern in it to find all the zipcodes in the file.

2) Create a scrapeable file for http://www.bananarepublic.com/customerService/storeLocator.do and set it to be scraped every time the first scrapeable file finds a zipcode.

3) Input the zipcode found from the file into the parameters of the 2nd scrapeable file.

4) Set extractor patterns in the 2nd scrapeable file to get all the store information you need.

Hope that helps.

-Alan

Can you give me the URL to the site you want to scrape? When I put the 206.231.92.100/ URL in my browser I get an error message, so could you tell me the exact URL of where you're trying to extract data?

Thanks,
Alan

Hi,

I want to extract the URL's (of the individual details pages) of all the companies listed when i search for a particular company from here. That is, for example, I want to search for 'european investment corporation'. Now there are 7 companies listed, and each one is a link to a page that gives details about that particular company. What I want is the URL of that details page. Now the way the URL is constructed makes it impossible for me to get the URL. First off, there is cookie setting to do, but this can be done. Then, there are a bunch of Javascript functions which has got a lot to do with Random number generation. These make further calls to other functions which obfuscate the parameters further. Is there anyway I can work around these?

Thanks and regards,
hemanth