Analytics
When you scrape a site with screen-scraper, will it cause a spike in the analytics for the site you are scraping? For example, I know if you scan a site using Xenu to get all the pages on a site, this does not show up in analytics (google analytics). Likewise, when other spiders come through a site, they are also not usually recorded by analytics. So how does screen-scroper behave, when I scan a site will it cause a huge spike in recorded traffic?
Thanks.
Depends on how they're
Depends on how they're reading their traffic. For google analytics, or any other type using Javascript, screen-scraper will not leave any footprints, because it doesn't run any javascript. It only requests raw HTML.
Usually this is good :P
If they actually look at their connections, they might see a flagrantly hammering IP address requesting pages faster than a human could possibly be clicking. We've gotten ourselves blocked on a few sites for that, haha..
So it depends. No, SS won't fire javascript. Yes, your sessions are still pretty visible to the site's admin. SS reports itself with a Mozilla user-agent, so it doesn't really stand out as anything strange. That being the case, though, you will be getting the HTML intended for the end-user, and the site should have no reason to divert you to the "robots.txt" no-indexing list.
Thanks
That is exactly what I was looking for... thanks for the quick response.