Lost on why it will not go to results page
Hi all-
I am working on a scrape for a client and can not figure out why the scrape will not go to the final results page. I figure because it goes through the scrape so fast. If you go on the website and submit the form it will show query in progress for a few seconds then take you to the results page where the table information is. I am lost why it will not pause and wait for the results page. Attached is the scrape that I built.
Jas
| Attachment | Size |
|---|---|
| B revised.sss | 29.81 KB |
| b fixed (Scraping Session).sss | 29.91 KB |
When you see that "waiting"
When you see that "waiting" page, there is either a meta tag or javascript sending an HTTP request every few seconds to see if the results are ready. Usually it's the same HTTP request as what spawned the timer. What you need to do is scrape something on the time page to let you know that it is indeed the timer, and if it is there, pause a couple of seconds and send the request again.
Jason- Thanks for the reply.
Jason-
Thanks for the reply. I looked through the javascript and the rest of the HTML, but could not find anything. Any other ideas?
Jasons, I've gone ahead and
Jasons,
I've gone ahead and replaced the sample scraping session with one that does not contain the login credentials. Please remember this is a public forum accessible to any and all.
Thanks,
Scott
Thanks Scott, I forgot to
Thanks Scott, I forgot to remove them before posting it.
Jason I went through it again
Jason
I went through it again line by line, but could not find anything where it calls it in the javascript. Can you see anything?
Thanks
Jason
Jason, So, it turned out that
Jason,
So, it turned out that all it needed was the latest PriceQueryRequestId and a little time to complete the processing. See scrape attached to the thread and test it under different scenarios because it may need some tweaking depending on what is sent in the search.
-Scott