POST data exceeded maximum length and was truncated
http://www1.dhcr.state.ny.us/BuildingSearch/Default.aspx is the site I want to scrape.The search has to be made using zip code.
In the page for making zipcode search there are two dropdown lists; one for county and one for zipcode for each county there is a list of zipcodes. I want the results for all of them.
First I created a scrapeable file to fetch the zipcode for a selected county .It ran smoothly. The next task was to scrape the data. I created a scrapeable file and when I ran the session the page is been redirected to the asperror page. I felt that the problem is with the viewstate. There could be a viewstate for each if the search.There only 1 page for each search. How can we handle such a situation if it is so?
Do I have to create a scrapeable file for each search? Can't do it with a single file changing the county and zipcode values?
I would like to get an immediate response.
thank you in advance.
regards,
BIBY
NB: In the request tab of the proxy for the search resut URL, I saw this message (POST data exceeded maximum length and was truncated) given at the end of the text. Is this the problem for the page not been scrapped ?
You'll have to scrape the
You'll have to scrape the VIEWSTATE variable from the previous page and put a variable in over top of your scrapeableFile's VIEWSTATE post parameter. It says that it's truncated because those kinds of sites are stupidly obsessed with having mile-long viewstates (which is completely needless, but that's their fault).
The viewstate would change each page request, so you'll have to scrape it on each page.
POST data exceeded maximum length and was truncated
I have a similar problem. I did exactly what you said i. e. scraped the VIEWSTATE variable from the previous page, saved it into session variable (say, ~@PARAM1@!~) and put it as a post parameter in the next file )as a ~#PARAM1#~). I've got a familiar notice in the last request tab:POST data exceeded maximum length and was truncated. Is there any way to lift the limit on the size of the POST parameters?
Thanks!
zfin, Whenever screen-scraper
zfin,
Whenever screen-scraper truncates data it is done for one reason: To allow the workbench interface to more easily render. The actual data being truncated in GUI remains untouched for interactions between the client and the server.
We made this change to screen-scraper's workbench after we found that when data reached a certain size it would cause screen-scraper's interface to freeze and become unresponsive. However, screen-scraper has no problem dealing with the large amounts of data underneath the GUI where all of the data is retained in it's original state.
If you need to see the entire content you can do the following.
Last Request: View in active log pane.
Last Response: Display in browser & view source.
Parameters: Display raw request from original proxy transaction.
A bit of advice on scraping .net sites...don't worry about what the viewstate looks like, just be sure to extract it from the page that preceded it when you proxied the site. Also, make sure it is the correct referer and manually set the referer if the sequence of your scraping session doesn't match the sequence expected by the server.
http://community.screen-scraper.com/API/setReferer
http://blog.screen-scraper.com/2008/06/04/scraping-aspnet-sites/
-Scott