Improve scrape speed
Hi,
I setup a scraping session that parses a relatively simple web page and returns like 100 rows of xml content . 20 nodes of 4 elements. say 20 pattern matches. Now I called scrape from C# using a RemoteScrapingSession object. I debugged the code and noticed that running the scrape command on this object takes about 10-12s. The machine is a 4 quad 3ghz. I'm in EU and scraping a US site.
Now, what can I do to improve the scrape time\ retrieve data faster for user?
See this blog post for some
See this blog post for some general notes:
http://blog.screen-scraper.com/2010/10/11/scaling-optimizing-screen-scraper/