Scraping ASP.NET site - loosing Additional Search Options page
I am attempting to rewrite my scripts to scrape a website that recently changed to asp.net. I am able to pull the EVENTTARGET, EVENTARGUMENT, and VIEWSTATE parameters and then post to the next page but when I get to the ERecruit page to select the Additional Search Options these parameters stop being passed/listed. I'm not quite sure if I need to us the PS_TOKEN or other variable set in the html but even when I try to use these I still am not able to pull up the Additional Search Options page from the ERecruit Search Page.
Now when I try it again I am getting the following response from the Home Page of http://www.aegonins.com/ :
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="generator" content="HTML Tidy, see www.w3.org" />
<title></title>
</head>
<body>
<table>
<tr>
<td></td>
<td><img src="images/errortitle.jpg" /></td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td></td>
</tr>
<tr>
<td></td>
<td><b>We're sorry but an error has occurred...</b></td>
</tr>
<tr>
<td> </td>
</tr>
<tr>
<td></td>
<td>The error has been logged and an e-mail has been sent to the appropriate support personnel.</td>
</tr>
<tr>
<td> </td>
</tr>
<tr>
<td></td>
<td>Click <a id="HomeURL" href="default.aspx">here</a> to return to the Home page.</td>
</tr>
</table>
</body>
</html>
The ER Basic Search Page is being displayed correctly but I am unable to access the following page which should be the Additional Search Options. Instead I get a very generic version of the ER Basic Search Page. Please help me, I know I am missing parameters here but I am just not finding them.
I can send the scraping session if needed but I need to get this resolved asap. Please email me with any questions or suggestions.
Thank you
Terribly sorry for the long
Terribly sorry for the long delay in response.
These asp.net pages can be a huge headache... It's a shame they're so popular.
It seems a bit tidier than other asp pages I've seen, but I'm notcing that it has javascript functions that aren't doing anything... I might try to find where a javascript function named "Page_ClientValidate" is located. It seems like the submit button has to run it first, before continuing. I'm not sure which page you were specifically looking at. I've been viewing the plain "search" page from the home page you gave me.
Not sure if that gives you any additional direction or not.. Hope it helps somehow, though.
Tim