Now, as to having screen-scraper perform the extraction instead of my cludgy combination of sample Google API Javascript & PHP, well... I'm sure it is possible, but probably not necessary since so much of the heavy lifting is being handled by Javascript. When asked to scrape a site where the data is made available via an API it is really a coin toss as to whether we will use screen-scraper for the job since each request and response is expected to be very structured.
Hope this helps.
-Scott
btw, I just noticed recently that you can view the panorama's in ThreeDee!!
Since my last post on the above subject I've used a different source for the photos I want, however the photos I was using was inferior to Google's Maps. As a result, I've decided to SS Google Maps, since I'm not familiar with API.
The photo I want is the front of a property and the data I have to retrieve it is an address and a zip code. I cleared my cookies and started the Proxy Session. I must say, I'm always thrown of how to procede when the data I inputed in the website does not show-up in the first couple of HTTP Transactions in the Progress of the Proxy Session. For example, I input 1413 W. 60th Pl. 90047, however the HTTP Transactions are as follows:
I recognize the data in the second HTTP Transaction as Long and Lat but I don't know how it was generated.
Since I didn't really know where to begin I generated scapeable files from the first 8 HTTP Transactions and then ran the scraping session. The results are as follows:
Starting scraper.
Running scraping session: Google Maps
Processing scripts before scraping session begins.
Scraping file: "File from New Proxy Session1"
File from New Proxy Session1: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session1: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0
File from New Proxy Session1: Sending request.
File from New Proxy Session1: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session11"
File from New Proxy Session11: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session11: Resolved URL: http://maps.google.com/maps/suggest?q=1&cp=1&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._15ggx44tvn
Setting referer to: http://gg.google.com/csi
File from New Proxy Session11: Sending request.
Scraping file: "File from New Proxy Session12"
File from New Proxy Session12: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session12: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.187%2Csgps0.187%2Csgps1.187%2Cacm1.187%2Cacu1.250
Setting referer to: http://maps.google.com/maps/suggest?q=1&cp=1&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._15ggx44tvn
File from New Proxy Session12: Sending request.
File from New Proxy Session12: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session13"
File from New Proxy Session13: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session13: Resolved URL: http://maps.google.com/maps/suggest?q=14&cp=2&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._16ggx44u4b
Setting referer to: http://gg.google.com/csi
File from New Proxy Session13: Sending request.
Scraping file: "File from New Proxy Session14"
File from New Proxy Session14: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session14: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.78%2Csgps0.78%2Csgps1.78%2Cacm1.78%2Cacu1.94
Setting referer to: http://maps.google.com/maps/suggest?q=14&cp=2&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._16ggx44u4b
File from New Proxy Session14: Sending request.
File from New Proxy Session14: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session15"
File from New Proxy Session15: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session15: Resolved URL: http://maps.google.com/maps/suggest?q=141&cp=3&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._17ggx44uf6
Setting referer to: http://gg.google.com/csi
File from New Proxy Session15: Sending request.
Scraping file: "File from New Proxy Session16"
File from New Proxy Session16: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session16: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.94%2Csgps0.94%2Csgps1.94%2Cacm1.94%2Cacu1.109
Setting referer to: http://maps.google.com/maps/suggest?q=141&cp=3&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._17ggx44uf6
File from New Proxy Session16: Sending request.
File from New Proxy Session16: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session17"
File from New Proxy Session17: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session17: Resolved URL: http://maps.google.com/maps/suggest?q=1413&cp=4&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._18ggx44unu
Setting referer to: http://gg.google.com/csi
File from New Proxy Session17: Sending request.
Processing scripts after scraping session has ended.
Scraping session "Google Maps" finished.
As you can see I received the following error:
File from New Proxy Session1: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
I think if you can point me in the right direction for the first part of SS Google Maps I can figure out the second part.
I'm not sure this is something that can feasibly be done using screen-scraper. Because Google's API is meant to be accessed via a browser there are certain Javascript calls that require the existence of a browser window. screen-scraper's built-in "browser" is Apache's HTTPClient which has no window object.
As an alternative I would recommend cooking up something using your favorite web programming language such as PHP or .NET.
Web Data Extraction
Adrian, Funny you'd ask.
Adrian,
Funny you'd ask. I've been researching this for a few months. The answer, in short, is sort of. Here's a summary of what I've found.
Intro
More...
Some more...
Some fun with it all...
Now, as to having screen-scraper perform the extraction instead of my cludgy combination of sample Google API Javascript & PHP, well... I'm sure it is possible, but probably not necessary since so much of the heavy lifting is being handled by Javascript. When asked to scrape a site where the data is made available via an API it is really a coin toss as to whether we will use screen-scraper for the job since each request and response is expected to be very structured.
Hope this helps.
-Scott
btw, I just noticed recently that you can view the panorama's in ThreeDee!!
Google Maps Revisited
Hello Scott,
Since my last post on the above subject I've used a different source for the photos I want, however the photos I was using was inferior to Google's Maps. As a result, I've decided to SS Google Maps, since I'm not familiar with API.
The photo I want is the front of a property and the data I have to retrieve it is an address and a zip code. I cleared my cookies and started the Proxy Session. I must say, I'm always thrown of how to procede when the data I inputed in the website does not show-up in the first couple of HTTP Transactions in the Progress of the Proxy Session. For example, I input 1413 W. 60th Pl. 90047, however the HTTP Transactions are as follows:
2 http://maps.google.com/maps/suggest?q=1&cp=1&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._15ggx44tvn Transaction complete.
3 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.187,sgps0.187,sgps1.187,acm1.187,acu1.250 Transaction complete.
4 http://maps.google.com/maps/suggest?q=14&cp=2&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._16ggx44u4b Transaction complete.
5 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.94 Transaction complete.
6 http://maps.google.com/maps/suggest?q=141&cp=3&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._17ggx44uf6 Transaction complete.
7 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.94,sgps0.94,sgps1.94,acm1.94,acu1.109 Transaction complete.
8 http://maps.google.com/maps/suggest?q=1413&cp=4&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._18ggx44unu Transaction complete.
9 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.79,sgps0.79,sgps1.79,acm1.94,acu1.110 Transaction complete.
10 http://maps.google.com/maps/suggest?q=1413+&cp=5&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._19ggx44wnw Transaction complete.
11 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.110,sgps0.110,sgps1.110,acm1.110,acu1.141 Transaction complete.
12 http://maps.google.com/maps/suggest?q=1413+w&cp=6&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1aggx44x7f Transaction complete.
13 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.110,sgps0.110,sgps1.110,acm1.110,acu1.125 Transaction complete.
14 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,acm1.0,acu1.16 Transaction complete.
15 http://maps.google.com/maps/suggest?q=1413+W&cp=6&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1bggx44zoe Transaction complete.
16 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.94,sgps0.94,sgps1.94,acm1.94,acu1.110 Transaction complete.
17 http://maps.google.com/maps/suggest?q=1413+W.&cp=7&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1cggx450of Transaction complete.
18 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.94 Transaction complete.
19 http://maps.google.com/maps/suggest?q=1413+W.+&cp=8&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1dggx450x4 Transaction complete.
20 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.94 Transaction complete.
21 http://maps.google.com/maps/suggest?q=1413+W.+6&cp=9&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1eggx4518e Transaction complete.
22 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.109,sgps0.109,sgps1.109,acm1.109,acu1.125 Transaction complete.
23 http://maps.google.com/maps/suggest?q=1413+W.+60&cp=10&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1fggx451g7 Transaction complete.
24 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.94,acu1.110 Transaction complete.
25 http://maps.google.com/maps/suggest?q=1413+W.+60t&cp=11&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1gggx4522c Transaction complete.
26 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.94,acu1.110 Transaction complete.
27 http://maps.google.com/maps/suggest?q=1413+W.+60th&cp=12&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1hggx4525t Transaction complete.
28 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.110,sgps0.110,sgps1.110,acm1.110,acu1.125 Transaction complete.
29 http://maps.google.com/maps/suggest?q=1413+W.+60th+&cp=13&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1iggx452ri Transaction complete.
30 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.79,sgps0.79,sgps1.79,acm1.79,acu1.94 Transaction complete.
31 http://maps.google.com/maps/suggest?q=1413+W.+60th+P&cp=14&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1jggx4538g Transaction complete.
32 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.94 Transaction complete.
33 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl&cp=15&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1kggx453gp Transaction complete.
34 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.93,sgps0.93,sgps1.93,acm1.93,acu1.109 Transaction complete.
35 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.&cp=16&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1lggx454b2 Transaction complete.
36 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.79,sgps0.79,sgps1.79,acm1.79,acu1.94 Transaction complete.
37 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,&cp=17&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1mggx4562v Transaction complete.
38 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.94 Transaction complete.
39 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+&cp=18&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1nggx456bk Transaction complete.
40 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.94,acm1.94,acu1.109 Transaction complete.
41 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+9&cp=19&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1oggx45aha Transaction complete.
42 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.94,sgps0.94,sgps1.94,acm1.94,acu1.110 Transaction complete.
43 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+90&cp=20&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1pggx45apj Transaction complete.
44 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.94,sgps0.94,sgps1.94,acm1.94,acu1.110 Transaction complete.
45 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+900&cp=21&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1qggx45atg Transaction complete.
46 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.16,xdc0.16,xdc1.94,sgps0.94,sgps1.110,acm1.110,acu1.125 Transaction complete.
47 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+9004&cp=22&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1rggx45beq Transaction complete.
48 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.78,sgps0.78,sgps1.78,acm1.78,acu1.93 Transaction complete.
49 http://maps.google.com/maps/suggest?q=1413+W.+60th+Pl.,+90047&cp=23&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&v=2&json=a&auth=638943:Zcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._1sggx45bhr Transaction complete.
50 http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0,acm0.0,xdc0.0,xdc1.94,sgps0.94,sgps1.94,acm1.94,acu1.109 Transaction complete.
51 http://maps.google.com/maps?f=q&source=s_q&output=js&hl=en&geocode=&abauth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&q=1413+W.+60th+Pl.%2C+90047&btnG=Search+Maps&aq=&vps=2&jsv=298d&sll=33.982846%2C-118.29604&sspn=0.008007%2C0.013078&g=6211+S.+Budlong+Ave+90044 Transaction complete.
52 http://maps.google.com/maps/gen_204/sgcd?source=s_q&ll=33.982846,-118.29604&spn=0.008007,0.013078&hl=en&gl=us&sgcd3=%5B%5B%5B,,1,,,%5B250,%5B%5B1,2,2,856338%5D,%5B1,2,3,17%5D%5D%5D,null%5D,%5B62,,1,,,%5B94,%5B%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,2,5140%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B297,,1,,,%5B109,%5B%5B1,2,1,177%5D,%5B1,2,1,17%5D,%5B1,2,2,177%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B203,,1,,,%5B110,%5B%5B1,2,1,17%5D,%5B1,2,4,177%5D%5D%5D,null%5D,%5B2500,,1,,,%5B125,%5B%5B1,2,1,17%5D,%5B1,2,2,177%5D,%5B1,2,1,618%5D,%5B1,2,1,177%5D%5D%5D,null%5D,%5B562,,1,,,%5B125,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B2594,,-1,,,%5B0,%5B%5B1,2,1,17%5D,%5B1,2,2,177%5D,%5B1,2,1,618%5D,%5B1,2,1,177%5D%5D%5D,null%5D,%5B0,,0,,,,null%5D,%5B484,,1,,,%5B110,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B1187,,1,,,%5B94,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B219,,1,,,%5B94,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B328,,1,,,%5B109,%5B%5B1,2,1,4386%5D,%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,2,17%5D%5D%5D,null%5D,%5B156,,1,,,%5B110,%5B%5B1,2,3,17%5D,%5B1,2,2,4385%5D%5D%5D,null%5D,%5B687,,1,,,%5B110,%5B%5B1,2,3,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B15,,1,,,%5B125,%5B%5B1,2,3,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B656,,1,,,%5B94,%5B%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B516,,1,,,%5B94,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B203,,1,,,%5B109,%5B%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B984,,1,,,%5B94,%5B%5B1,2,3,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B2203,,1,,,%5B94,%5B%5B1,2,3,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B219,,1,,,%5B109,%5B%5B1,2,3,17%5D,%5B1,1,1,3347%5D,%5B1,2,1,17%5D%5D%5D,null%5D,%5B5281,,1,,,%5B110,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B203,,1,,,%5B94,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B31,,1,,,%5B109,%5B%5B1,2,5,17%5D%5D%5D,null%5D,%5B657,,1,,,%5B93,%5B%5B1,2,2,17%5D%5D%5D,null%5D,%5B32,,1,%221413+W.+60th+Pl.,+90047%22,,%5B93,%5B%5B1,2,2,17%5D%5D%5D,null%5D%5D,1000,1000,,%5B1%5D%5D Transaction complete.
53 http://maps.google.com/maps/stk/geocodes?ll=33.982851%2C-118.296061&spn=0.008007%2C0.013078&z=17&hl=en&format=p&gcs=Los%20Angeles&gr=CA&gl=US&ftid=0x80c2b7d8f1e30887%3A0x93aa118d10fd0f76 Transaction complete.
54 http://maps.google.com/favicon.ico Error.
55 http://maps.gstatic.com/intl/en_us/mapfiles/298d/maps2/mod_rv.js Transaction complete.
56 http://mt0.google.com/vt/ft?lyrs=m%40139&las=tuwtvuwvvwtvvwvuw,tuwtvuwvvwtvvwwtv,tuwtvuwvvwtvvwwtw,tuwtvuwvvwtvvwwuv,tuwtvuwvvwtvvwwuw,tuwtvuwvvwtvvwvwu,tuwtvuwvvwtvvwwvt,tuwtvuwvvwtvvwwvu,tuwtvuwvvwtvvwwwt,tuwtvuwvvwtvvwwwu,tuwtvuwvvwtvvwvww,tuwtvuwvvwvttutuu,tuwtvuwvvwvttutuw,tuwtvuwvvwvttutwu&z=17&gl=us&hl=en&xc=1&opts=z&callback=_xdc_._1tggx45gk1 Transaction complete.
57 http://mt0.google.com/vt/ft?lyrs=m%40139&las=tuwtvuwvvwtvvwvuw,tuwtvuwvvwtvvwwtv,tuwtvuwvvwtvvwwtw,tuwtvuwvvwtvvwwuv,tuwtvuwvvwtvvwwuw,tuwtvuwvvwtvvwvwu,tuwtvuwvvwtvvwwvt,tuwtvuwvvwtvvwwvu,tuwtvuwvvwtvvwwwt,tuwtvuwvvwtvvwwwu,tuwtvuwvvwtvvwvww,tuwtvuwvvwvttutuu,tuwtvuwvvwvttutuw,tuwtvuwvvwvttutwu&z=17&gl=us&hl=en&xc=1&opts=z&callback=_xdc_._1uggx45gqz Transaction complete.
58 http://maps.google.com/maps/gen_204?imp=smimps%3D2NZBnlvpod,HXkBjPnO7IR,GNh1I4a0Lkl,Jk6ybLkezLR,Igw4mrxKfsr,EtLo6v5eTmV,K8w0JZcw1U6,DTPCDdz9acL,MkUaYUiIHWi,BSiVnShBAOC%26z%3D17&ei=2t7tTJmnO5GIiAPH353RBQ Transaction complete.
59 http://gg.google.com/csi?v=3&s=mfe&action=vpage&e=200864&cad=vpm:appiw|mssvt|actbr|lt_j|strr|adf|act_s|act,nmkr:1,mt:mhm,dup:ft1,aa:1,actvp:1&rt=vp0.0,vp1.375,vpcps.375,vpsv0.375,vpsv1.390,vplm0.422,vplm1.422,vpp0.422,mkr0.422,mkr1.437,cpxd.437,xdc0.469,afvp.469,vpp1.484,sact.531,xdc1.750,sid.2687,ft0.3015,pbd.2797,pxd.2797,plt.2797,pdt.2797,ft1.3703 Transaction complete.
60 http://fxfeeds.mozilla.com/en-US/firefox/headlines.xml Transaction complete.
I recognize the data in the second HTTP Transaction as Long and Lat but I don't know how it was generated.
Since I didn't really know where to begin I generated scapeable files from the first 8 HTTP Transactions and then ran the scraping session. The results are as follows:
Running scraping session: Google Maps
Processing scripts before scraping session begins.
Scraping file: "File from New Proxy Session1"
File from New Proxy Session1: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session1: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0
File from New Proxy Session1: Sending request.
File from New Proxy Session1: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session11"
File from New Proxy Session11: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session11: Resolved URL: http://maps.google.com/maps/suggest?q=1&cp=1&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._15ggx44tvn
Setting referer to: http://gg.google.com/csi
File from New Proxy Session11: Sending request.
Scraping file: "File from New Proxy Session12"
File from New Proxy Session12: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session12: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.187%2Csgps0.187%2Csgps1.187%2Cacm1.187%2Cacu1.250
Setting referer to: http://maps.google.com/maps/suggest?q=1&cp=1&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._15ggx44tvn
File from New Proxy Session12: Sending request.
File from New Proxy Session12: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session13"
File from New Proxy Session13: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session13: Resolved URL: http://maps.google.com/maps/suggest?q=14&cp=2&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._16ggx44u4b
Setting referer to: http://gg.google.com/csi
File from New Proxy Session13: Sending request.
Scraping file: "File from New Proxy Session14"
File from New Proxy Session14: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session14: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.78%2Csgps0.78%2Csgps1.78%2Cacm1.78%2Cacu1.94
Setting referer to: http://maps.google.com/maps/suggest?q=14&cp=2&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._16ggx44u4b
File from New Proxy Session14: Sending request.
File from New Proxy Session14: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session15"
File from New Proxy Session15: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session15: Resolved URL: http://maps.google.com/maps/suggest?q=141&cp=3&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._17ggx44uf6
Setting referer to: http://gg.google.com/csi
File from New Proxy Session15: Sending request.
Scraping file: "File from New Proxy Session16"
File from New Proxy Session16: Preliminary URL: http://gg.google.com/csi
File from New Proxy Session16: Resolved URL: http://gg.google.com/csi?v=3&s=mfe&action=suggest2&e=200864&rt=acu0.0%2Cacm0.0%2Cxdc0.0%2Cxdc1.94%2Csgps0.94%2Csgps1.94%2Cacm1.94%2Cacu1.109
Setting referer to: http://maps.google.com/maps/suggest?q=141&cp=3&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._17ggx44uf6
File from New Proxy Session16: Sending request.
File from New Proxy Session16: An error occurred while getting the body of the HTTP response. The message was: HTTP entity may not be null.
Scraping file: "File from New Proxy Session17"
File from New Proxy Session17: Preliminary URL: http://maps.google.com/maps/suggest
File from New Proxy Session17: Resolved URL: http://maps.google.com/maps/suggest?q=1413&cp=4&ll=33.982846%2C-118.29604&spn=0.008007%2C0.013078&hl=en&gl=us&v=2&json=a&auth=638943%3AZcz9Tc6Lje4iOJpcNgaKDwY26is&src=1&num=5&numps=5&clid=1&callback=_xdc_._18ggx44unu
Setting referer to: http://gg.google.com/csi
File from New Proxy Session17: Sending request.
Processing scripts after scraping session has ended.
Scraping session "Google Maps" finished.
As you can see I received the following error:
I think if you can point me in the right direction for the first part of SS Google Maps I can figure out the second part.
Thanks for all your help and happy turkey-day.
AdrianJay
Adrian, I'm not sure this is
Adrian,
I'm not sure this is something that can feasibly be done using screen-scraper. Because Google's API is meant to be accessed via a browser there are certain Javascript calls that require the existence of a browser window. screen-scraper's built-in "browser" is Apache's HTTPClient which has no window object.
As an alternative I would recommend cooking up something using your favorite web programming language such as PHP or .NET.
The basic steps are straight-forward:
1. Query an address
2. Retrieve latitude and longitude
3. Query for nearest panoid
4. If found, download corresponding image tiles and compile into final image using Imagemagick's montage
I hope this helps,
-Scott