Home » Forums » screen-scraper » screen-scraper public support

login to flickr

Hello

i want to login to flickr in order to scrape some pictures, but it always fails to do so.

http://www.flickr.com

Anybody did this before?

So i had the idea it would be good to be able to login manually and then continue.

So when its time to login i want screen scraper to open a browser - i login manually - then screen scraper should continue with me logged in.

Is that possible?

Regards

Ben

‹ Sorry, tidying HTML failed. Returning the original HTML MaxConcurrentScrapingSessions and Server.NumTimesRun properties ›
nebben on 04/02/2012 at 11:26 pm

I don't have a way for

I don't have a way for screen-scraper to interact with a browser. What you'd need to do is make a script that has the credentials, or a script that reads in a file that has the credentials. See tutorial #2 for some instructions.

 jason on 04/03/2012 at 9:20 am

tutorial #2 is not helpful

what do you mean with writing a script that has the credentials?

in tutorial #2 login works with the parameters tab, where you insert the crendentials to post.

But this does not do it at flickr.

so what can you do?

 nebben on 04/04/2012 at 8:59 am

I did a quick proxy, and used

I did a quick proxy, and used the Yahoo login. The credentials are on a page https://login.yahoo.com/config/login_verify2 in the POST parameters.

 jason on 04/04/2012 at 9:18 am

thanks i did the same, but

thanks i did the same, but after that you are not logged in at least i am not.

are you?

plese try

http://www.flickr.com/groups/geology/members/

and see if you get it or if you are refered to the login again

 nebben on 04/04/2012 at 11:11 pm

nebben, After proxying the

nebben,

After proxying the site I notice a number of potential transactions that you'll likely need to turn into scrapeable files. These are the base URLs (minus any post parameters) of those I think you'll want to account for.

  1. (start)
    http://www.flickr.com/
  2. (302 automatic redirect)
    http://www.flickr.com/signin/
  3. (this is where it redirects to)
    https://login.yahoo.com/config/login
  4. (this is where you submit your credentials)
    https://login.yahoo.com/config/login
  5. (this page is loaded per some Javascript, I believe)
    https://login.yahoo.com/config/validate?.src=flickrsignin&.pc=8190&.scrumb=8ayBQgx3bUT&.pd=c%3DJvVF95K62e6PzdPu7MBv2V8-&.intl=us&.done=http%3A%2F%2Fwww.flickr.com%2Fsignin%2Fyahoo%2F%3Fredir%3Dhttp%253A%252F%252Fwww.flickr.com%252F
  6. (302 this is where it redirects to and this also redirects again)
    http://www.flickr.com/signin/yahoo/?redir=http%3A%2F%2Fwww.flickr.com%2F&.data=<long string here>&.ys=Jwn3dgSzsZrBWXqZiFxcVl.Jmf4-
  7. (302 this is where it redirects to and this redirects once again)
    http://www.flickr.com/cookie_check.gne?pass=http%3A%2F%2Fwww.flickr.com%2F&fail=register_cookies.gne
  8. (now you're in)
    http://www.flickr.com/

Of the above, you may only need to handle passing the correct data in either the POST or GET for the non-302 items. screen-scraper handles all 302's automatically (including the data sent in the POST or GET because that data is generated on the server and not on the client).

#5 could be the one that you need to pay the closest attention to.

Hope this helps,
Scott

 swilsonmc on 04/06/2012 at 3:13 pm