At the end of my time at Zipfian Academy (an intensive data-science program), we were given two weeks to design and implement a data-science project. I wanted my project to elicit interesting and even actionable insights about a topic that matters to a lot of people, and I also wanted the project to have, if developed further, some cool applications.

I decided to build a neighborhood recommender for San Francisco. If you are looking to buy or rent a place, it can often be difficult, especially in SF, to find availability in the neighborhood you want. It can be even more difficult to find a place that fits within your budget. To build a “hood” recommender, I first found geographically tagged quantitative data on events and places in San Francisco. I then merged these data sources together and computed a similarity matrix of SF hoods, indexing how similar every hood in SF is to every other hood. This similarity matrix can be used to help people find alternative neighborhoods, particularly off-the-beaten-path hoods, that are similar to known and desirable ones.

Building a model of SF ‘hood similarities

As a first step, I collected geotagged datasets from a variety of sources:

SF’s open database (quality-of-life complaints, crime reports, # of restaurants, business types & opening dates) Trulia’s API (# of bedrooms in houses for sale, housing prices) Walkscores.com (transportation density) Google (fog scores). A set of polygons that partitioned SF into 117 neighborhoods.

For data munging and analysis, I used the following libraries in Python, including:

Shapely - binned data points into their respective neighborhoods and to measure the area of a neighborhood Pandas - categorized data by time; removed poor features (e.g., features composed mostly of 0’s) and hoods (e.g., parks); merged data files and transformed features Scikit-Learn - scaled and imputed data; reduced redundant features with PCA; created a similarity matrix from features, and visualized similarity matrix Matplotlib - for data visualizations

Validating the model with domain knowledge

Based on my knowledge of SF, the similarity matrix made very good estimates of neighborhood similarities. For example, the top-5 hoods that are the most similar to the Mission District are:

Mission Dolores South of Market Lower Haight Lower Nob Hill Tenderloin.

If you know the city, you know that these hoods are very similar to the Mission. Neighborhoods that are similar to the Mission are visualized below with the blue star marking the Mission. Light colored areas are similar to the Mission and dark-red areas are less similar.

Fun Insight: Noe Valley is a poor man’s Russian Hill.

The model also found Noe Valley, a rich neighborhood with older denizens near the Mission District, to be similar to a set of other particularly rich neighborhoods with older denizens. (See the figure below) The top-5 most similar hoods to Noe Vally are:

Ashbury Heights Cow Hollow Holly Park Russian Hill Peralta Heights

Interestingly, the median price of three-bedroom houses in Noe Valley ($1.2 million) were 40% of what they are in a very similar neighborhood, Russian Hill ($3.1 million). And although Noe Valley may not have the same amazing views, my model suggests that Noe Valley offers a lot of what Russion Hill does. This means that it may be a good alternative for someone on a “budget.” It is of note that someone new to the city may know of Russian Hill because of its proximity to touristy areas, but they may not be familiar with an 'off-the-beaten-path’ neighborhood like Noe Valley. This is where a neighborhood recommender becomes pretty awesome.

For an overall assessment of the similarity matrix, I created the visualization below with multidimensional scaling. Neighborhoods that are more similar to each other will be closer to each other in this 2d space. The similarity matrix does an amazing job at assessing similarity. Click on image below.

ANALYSIS.

Which hoods, when compared to their top-5 most similar hoods, are abnormally cheap or expensive?

What if we compared the value of a neighborhood to its top 5 most similar hoods? Would it be cheaper or more expensive than its peers? What if we went through every neighborhood in the city, comparing the value of each hood to the average value of its top-5 most similar hoods? If we do this, we would obtain a value, the price difference for each hood, which can then be converted into z-scores to see if any of them are abnormally cheap or expensive (e.g., z-score greater than 2.5). These neighborhoods may be under- or over-valued and worthy of more exploration.

Following this approach, I first used housing prices per square foot as my measure of a neighborhood’s housing value. I found that a small, little-known neighborhood south of Visitacion Valley called Little Hollywood popped up as being abnormally cheap (z score > 2.5). Coincidentally, I just saw an article in SFgate that discusses how this little-known neighborhood is under appreciated (SFgate_article) - suggesting that my approach identified an off-the-beaten-path neighborhood that is of good value.

When examining housing prices in each neighborhood by the number of bedrooms in the house, Hunters Point showed up as being abnormally cheap compared to its peers. See the box plots of these data (made with matplotlib) by clicking on the image below.

So why are Little Hollywood and Hunters Point abnormally cheap?

Hunter’s point is notorious for its crime levels. Little Hollywood is a little neighborhood just south of another part of the city that is known for being dangerous, Visitacion Valley (also in the southern part of the city). Are these hoods cheaper than their peers because they are the most dangerous hoods in town? To check out this hypothesis, I pulled my crime data up and looked at where these hoods were ranked each year by crime type compared to other hoods in SF (i.e., from the #1 most dangerous crime-wise to 117th least dangerous). See the figure below. Little Hollywood does not seem like a dangerous hood; for total crime, it is ranked around 100/117. Amazingly, even though Hunters Point has a very bad reputation, it is nowhere near the top of the list for any type of crime. When looking at total crime it is near 100th place; although if crime does occur, it tends to be violent (~ 60th place; middle of the pack). Maybe Hunters Point no longer deserves its reputation as being dangerous. If the reputation of a neighborhood no longer fits the data, then the hood may be on the verge of some significant changes.

FUTURE APPLICATIONS.

This work has a number of applications. For example, any company that provides geographic services, such as Trulia, Airbnb, Craigslist, Priceline, etc. could use a similarity matrix like the one I built to automatically provide neighborhood recommendations to people who cannot find hotel or housing availability in a neighborhood they want and are aware of.

Similar data can also be found on cities throughout the country. It would therefore be possible to determine how similar hoods in SF are to hoods in other cities. Which neighborhoods in Austin or New York city are similar to Potrero Hill? To the Mission? To Noe Valley?

If you are a business owner and you want to expand your restaurant chain or retail store, you would want to know which hoods are similar to the one you have had success in. For example, if your retail store was successful in the Mission, you probably wouldn’t want to open up your second store in a hood that is similar to the Marina. A similarity matrix could be used to help business owners find similar hoods in a city of their choice or, alternatively, to mine all similar hoods across the U.S. to identify the similar hood that is likely to represent the best value.

EXTRAS

To see the code I wrote for the project, please see the ipython notebooks in my github account: Anson’s github

I used Python (Pandas, NumPy, skit-learn, shapely, matplotlib) for data munging, analysis, and most visualizations. I used CartoDb for the heat maps of hood similarities. An interactive D3 version is in the works.