Dumb question
Hi, this is probably a stupid question, but I am new to the world of creating websites having mostly done desktop apps in the past. Anyway I have screen scraper set to be invoked externally from my C# program, and everything works great with this desktop app, but I want to create a website now. I am just wondering how one normally uses screen screeper in this way, e.g. I have an ASP host, but they wont let me install screen scraper to the server for "security reasons", so how would I get my webpage to retrieve the data and where would I have screen scraper running through it's infinite set of scraping cycles?
Would I normally have to have the server run from my home machine, just updating an SQL database on my ASP host?or is there another way to host screen scraper?
Just wondering the route people would normally be taking with creating dynamic websites using screen scraper
Thanks again!
SQL to the rescue
Hi,
So it sounds like screen-scraper cannot be installed on the same machine that hosts your web page. In this case it is still simple to use screen-scraper to keep your website dynamic.
Here is my solution from start to finish:
Install screen-scraper on a machine you have admin access to. Once installed, develop the scrape or import the finished scrape. Next you will want to install MYSQL on the server that is hosting your website - Most hosting services will allow some flavor of SQL.
After SQL is installed on your hosting machine, develop a webpage using whatever languages your webserver allows you to load a page from. The page has to be able to have bindings for the database type. For example: PHP has MYSQL built into it. You will use GET or POST parameters to send information from screen-scraper to this website to insert those parameters into the hosting database.
Once the variables are inside the hosting database, it is a simple process to retrieve them for your webpage.
Just my 2 cents.
Best of luck
Hey, thanks for the reply
Hey, thanks for the reply Scraper. Yeah my ASP host wont let me install any external applications sadly (if anyone knows one that would allow me to install screen scraper, I'd be happy to change though?)
I would still need to have screen scraper runnining continously on the machine I have admin acess too(i.e. a home PC in my case) wouldnt I? Which is pretty sim to the way I'm doing it at the moment, having it loop on my home machine, update a local SQL db, then have the webpage connect to my home machine to access the db when it needs to....
Seems like a tricky one to me, I would have to buy a dedicated computer to be on 24/7 running the scrapes. There must be somewhere I can install screen scraper and have them host it?
time to switch hosts?
Great question.
A couple of points about hosting. If you do decide to install screen-scraper on a host, make sure it is a "dedicated" host a.k.a. you have your own box someplace on a server farm. These are a bit more pricy, but one that we are familiar with that will work is server beach. Their product works well with screen-scraper and they should give you the autonomy that you need to manage your own life - instead of having it managed for you. :) We've seen problems in the past working with hosting that is "virtually dedicated". If you need more info about this distinction send an email to [email protected] .
A little cheaper route would be to use Amazon EC2. It's a little more technical, but this also has worked in the past. You pretty much just rent a computer space from them (i'm pretty sure it is virtualized), but after that you install screen-scraper and get to work. They might charge you a little more if you need additional hard drive space, but I'd look into it.
Hopefully this helps.