bayarea.find().count()

"""
Explanation: Data Wrangling with OpenStreetMap <br/> Final Project
Author: William Truong
Map Area: San Francisco Bay Area, CA, United States
Sources:

Mapzen Metro Extract Link  
Audit and Shaping into JSON IPython Notebook  
Audit and Shaping into JSON IPython Notebook (html)

1. Problems Encountered in the Map
Several inconsistencies were discovered during the audit:

Concerning the "addr:street" key
Street names that don't share a common suffix ("1st Street vs. 1st St")
Inconsistent use of N vs North, S vs South, etc.
Address number included in the street names ("Adrian court, Suite A")
Intersections are used for some of the values (Pacheco Blvd @ Blum Rd N Of Sr 4)
Landmarks are used for some of the values (Tanforan Shopping Center).
Discernable amount of keys prefixed with tiger
Postal codes (95624 vs. 95624-6003.

"addr:street" conversion"
In the process of converting the "addr:street" keys into json under the "address"  name, street name suffixes were only fixed since the analysis that was performed did not concern street names in particular.
tiger data
There was a big number of keys prefixed with tiger. After some researching, it was found that tiger keys originate from Census data. OpenStreetMap initially used tiger data to import mass amounts of map information during the early years.  Future census data will no longer be imported and tiger data is being slowly converted to fit the recommended OpenStreetMap schema. Until it is properly converted, some of these key values need to be converted to the "address" dictionary.
Two keys were converted:
"tiger":"zip_left" -> "address":"postcode"
"tiger":"county" -> "address":"county"
These two keys were converted only if there wasn't a value in place for those key values located under the address dictionary.
2. Data Overview
File sizes
san-francisco-bay_california.osm .............. 1.9G
san-francisco-bay_california.osm.json ......... 2.2G
Number of documents
End of explanation
"""


bayarea.find({"type": "node"}).count()

"""
Explanation: Number of nodes
End of explanation
"""


bayarea.find({"type": "way"}).count()

"""
Explanation: Number of ways
End of explanation
"""


pipeline = [{"$match": {"amenity": {"$ne": None}}},
                       {"$group": {"_id": "$amenity",
                                   "count": {"$sum": 1}}},
                       {"$sort": {"count": -1}},
                       {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top 10 types of amenities
End of explanation
"""


# Top 10 fast food chains
pipeline = [{"$match": {"amenity": "fast_food", "name": {"$ne": None}}},
            {"$group": {"_id": "$name", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top 10 fast food chains
End of explanation
"""


pipeline = [{"$match": {"leisure": {"$exists": 1}}}, 
            {"$group": {"_id": "$leisure", "count": {"$sum": 1}}},
                       {"$sort": {"count": -1}},
                       {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top 10 types of leisurely activities
End of explanation
"""


pipeline = [{"$match": {"leisure": {"$exists": 1}, "address.city": {"$exists": 1}}},
            {"$group": {"_id": "$address.city", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top cities with the most leisurely activities
End of explanation
"""


pipeline = [{"$match": {"amenity": {"$exists": 1}, "address.city": {"$exists": 1}}},
            {"$group": {"_id": "$address.city", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top cities with the most amenities
End of explanation
"""


pipeline = [{"$match": {"building": {"$exists": 1}, "address.city": {"$exists": 1}}},
            {"$group": {"_id": "$address.city", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top cities with the most buildings
End of explanation
"""


pipeline = [{"$match": {"address.city": {"$exists": 1}}},
            {"$group": {"_id": "$address.city", "count": {"$sum": 1}}},
            {"$sort": {"count": -1}},
            {"$limit": 10}]
result = bayarea.aggregate(pipeline)
pprint(result)

"""
Explanation: Top cities with the most ways and nodes
End of explanation
"""


bayarea.find({"type": "node"}).count()

bayarea.find({"type": "node", "address.city": {"$exists": 0}}).count()

bayarea.find({"type": "node", "address.county": {"$exists": 0}}).count()

bayarea.find({"type": "node", "address.postcode": {"$exists": 0}}).count()

"""
Explanation: 3. Other Ideas About the Datasets
Cleaning Up OpenStreetMap Data
Open Street Map data is not only incomplete but also filled with inconsistencies and lack of structure. Some ideas need to be discussed at a public forum to organize OpenStreetMap data into a more structured form to make this public open source collaboration more relevant compared with Google Maps, Apple Maps, and Yelp.
Improvement 1: Fill out the city, county, zip code for all the nodes
End of explanation
"""


disp.Image("./images/leisure.png")

"""
Explanation: For the city, county, and postcode there is only < 1% coverage for all the nodes. During the shaping of the data to JSON, these values can be programmatically filled out if I had a geographical database that can fill out the city, county, and postcode from latitudinal and longitudinal coordinates. Once these values are completed, then more inferences can be made for city-city, county-county, or post code - post code comparisons.
It also would not hurt to recommend to OpenStreetMap to automatically fill these fields for a user when a node or way is inputted or editted.
Improvement 2: Making Key Values More Relevant
At this stage, I believe OpenStreetMap data for the bay area is too sparse and fragmented to be able to make solid inferences from it. If we take a look at the database for the types of values for the leisure keys:
End of explanation
"""


def css_styling():
    styles = open("../css/custom.css", "r").read()
    return disp.HTML(styles)
css_styling()

"""
Explanation: Source: http://taginfo.openstreetmap.org/keys/leisure#similar
Majority of the values for the leisure key fall under 6 values. That makes 1799 items that are irrelevant (with 1 item being < 1% of the total count) for making inferences. I believe there needs to be a discussion on how these 1799 values can be consolidated into groups. 
Benefits

Makes data more cross referenceable since nodes will be able to be related to data point in other datasets through city and/or county.
After consolidation of values for different keys, more inferences can be made and as a result data can be used in many opportunities

Potential Issues

Size of dataset will drastically increase due to the document oriented architecture of Open Street Map data.
How does an open source project reach a common consensus when consolidating information. Contributers are scattered accross the world.
Once consolidation of values for keys is achieved, how do we prevent it from being scattered again, while still providing a platform that is community friendly?

Anomaly: Ununsual amount of entries for Stockton
Analysis of the queries show an unusual amount of entries for the city of Stockton. The number of documents in Stockton is almost 11x the amount in San Francisco. More research needs to be done on why.
If there are incentives or someone has created an efficient script to input/modify data into OpenStreetMap from other sources for the city of Stockton, then we should find out what they are to give the tools more visibility.
Conclusion
The Open Street Map service is a valuable service that allows the community to rapidly update information about their surrounding areas. However, a common problem with big scale open collaboration efforts is that structure is often sacrificed with the vast amount of collaborators. There needs to be an open forum on how to combat the chaos that naturally comes with open collaboration to keep OpenStreetMap becoming irrelevant compared with Google/Apple maps, Yelp, etc. OpenStreetMap has enormous potential and it is up to the community to spearhead efforts to improve on it's infrastructure.
End of explanation
"""


