text:
tag:blogger.com,1999:blog-2357316514436369105.post6385728699480432684..comments2014-12-19T00:51:43.163ZComments on Theoretical Structural Archaeology: When on Google Earth 85Geoff Carterhttp://firstname.lastname@example.com:blogger.com,1999:blog-2357316514436369105.post-52037798731082867362010-03-09T13:11:25.260Z2010-03-09T13:11:25.260ZYou are most gracious, Geoff, thank you. I posted...You are most gracious, Geoff, thank you.

I posted the puzzling image here:
http://ancientworldbloggers.blogspot.com/2010/03/real-where-on-google-earth-86-following.htmlAlex O'Brienhttp://email@example.com:blogger.com,1999:blog-2357316514436369105.post-8868945382818999932010-03-09T10:11:03.387Z2010-03-09T10:11:03.387ZI am very sorry; I clearly acted in haste; I will ...I am very sorry; I clearly acted in haste; I will gladly take my WOGE85 post down post.
Please let me know what you want to do.Geoff Carterhttp://firstname.lastname@example.com:blogger.com,1999:blog-2357316514436369105.post-36645691115477607492010-03-09T03:11:48.589Z2010-03-09T03:11:48.589ZUm, I was about to post a new image. Please, bear ...Um, I was about to post a new image. Please, bear in mind we're not all in your time zone. It was only yesterday that I got it. The places I wanted to post turned out to not be very decipherable from Google earth so it took me a while to find something.
Also, it was the weekend, I have kids...Pojomumhttp://email@example.com:blogger.com,1999:blog-2357316514436369105.post-84452396581002613282010-03-08T19:49:41.268Z2010-03-08T19:49:41.268ZI will post a new WOGE[86], at 12pm local, unless ...I will post a new WOGE[86], at 12pm local, unless the previous winner comes forward, I hope this is OK . . .Geoff Carterhttp://email@example.com:blogger.com,1999:blog-2357316514436369105.post-30637191889067771092010-03-08T16:22:18.076Z2010-03-08T16:22:18.076ZHow about a 48-hr rule to post the next challenge?...How about a 48-hr rule to post the next challenge? Failure to do so allows the most recent host to re-post anew?Southieshamhttp://firstname.lastname@example.com:blogger.com,1999:blog-2357316514436369105.post-48031788814360070062010-03-08T12:50:56.266Z2010-03-08T12:50:56.266ZHi All, So far Pojomum has been a bit reluctant to...Hi All,
So far Pojomum has been a bit reluctant to take up their prize.

Please let us know if you need help, etc., or we might need a re-run to avoid breaking the chain.

Any thoughts?Geoff Carterhttp://email@example.com:blogger.com,1999:blog-2357316514436369105.post-28369674363565665932010-03-07T14:30:17.510Z2010-03-07T14:30:17.510ZWell Done, Pojomum, it is the Beautiful Bamburgh C...Well Done, Pojomum, it is the Beautiful Bamburgh Castle on the coast of Northumberland.
You have won the honour of hosting the next When on Google Earth.
Good luckGeoff Carterhttp://firstname.lastname@example.com:blogger.com,1999:blog-2357316514436369105.post-23076770875449202152010-03-07T14:00:13.575Z2010-03-07T14:00:13.575ZBamburgh Castle, Northumberland: while site has ea...Bamburgh Castle, Northumberland: while site has earlier occupation, the Norman castle was built by Henry II remaining largely impregnable for four centuries until becoming one of the first castles to succumb to cannon fire during the Wars of the Roses. The forces of the Earl of Warwick pounded it to pieces in 1464. What remains today is the Norman Keep.Pojomumhttp://email@example.com


text:
August cooled off and little or no rain here in Chianti.
The grapes are small, it will be interesting to see if little water raised the sugar content and hence a fabulous year for small but high quality production.
The fear factor with the weather like this is always hail near harvest time which can ruin the crops.
The other day we took an unknown road near us, a favorite passtime, getting lost on purpose
and ended up near a castle with a fabulous view of the " Crete Senese" the soil the color that inspired Crayola!
In our garden this year.. we were gifted by a little bird with a zucchini plant. It is slowly taking over the garden.
Since I have been down and out with a bad back, I haven't bent over yet to harvest fo zucchini blossoms to fry, but am feeling better so watch out!

subset: commoncrawl


text:
Sr. Software Engineer
Location : Detroit, MI
5+ years of programming or related experience in one or more of the following: Cocoa, Objective C, HTML, .NET, PHP, Java, or Progress
5+ years experience in software testing and design
5+ years of database related experience
Dec 22 -
Data Scientist
Develop and implement text mining and machine learning algorithms to improve the online user experience. Design and drive the collection of new, big data and refinement of existing data sources, standards and best practices.
· Report and analyze data ...
Dec 22 -
Social Media Advisor
Digital Chat Advisor - Detroit, MI - Contract
Company Description
Seed Staffing is finding the right soil for our seeds to flourish by connecting technology and humankind. As a tech-focused staffing company dedicated to helping technology professionals and companies ...
Software Release Coordinator
We are now seeking a Software Release Coordinator in the Redford, MI area! Trillium Technical, a leader in the staffing industry is currently seeking a Software Release Coordinator in the Redford, MI area!
Essential Duties and Responsibilities
• Coordination of software ...
Dec 22 -
Farmington Hills
Programmer Analyst
Renkim Corporation, an Information Services Company, founded in 1982 and located in Southgate Michigan, is seeking a full time Programmer Analyst.
Position Background:
Working in a fast paced team environment, successful candidates should be goal oriented and organized ...
Dec 22 -
Powertrain Controls Software Planning Consultant II
Powertrain Controls Software Planning (PCSP)
Our client has formed a small focused team to develop powertrain control programs on very compressed timing. The selected candidate will be comfortable with facilitating information between the PCSP group and Program teams, ...
Dec 22 -
.Net Developer
Long term contract
.Net development experience (particularly C#)
Microsoft Sql server development experience
Microsoft Access (preferable not a must)
Some networking and firewall experience will help very much
Mainframe knowledge and interface processing
First line ...
Dec 22 -
Auburn Hills
Back Office Systems Admin II
Responsible for supporting business hardware and software systems, this includes network and server environments along with storage and backup systems. Other responsibilities include maintenance of current ICT systems such as backups and OS patches and all ...
Business Systems Analyst
Supports and analyzes user requirements, procedures and problems to automate processing or to improve existing business system through standardization by performing the following duties.
(Include the following although other ...
Client Success Specialist - Level 3
This position is located within the Services and Support Department of 365 Retail Markets, and reports to the Client Success Manager. You will play a key role in ensuring that our clients receive a consistent and exceptionally positive experience at all times. Your primary ...
Dec 22 -
Senior Software Project Engineer Farmington Hills, MI
Senior Software Project Engineer
Job Location : Farmington Hills, MI
Job Description : provide safety systems such as Forward Collision Warning (FCW), Following Distance Indication (FDI), Lane Keeping assistance (LKAS) and Lane ...
Dec 21 -
Farmington Hills
Dec 21 -
Integrity SR Systems Administrator Farmington Hills, MI
Integrity SR Systems Administrator
Job Location : Farmington Hills, MI
Job Description : Responsibilities:
Integrity Client-, Integrity FSA Server-, Integrity Server- Administration, and system Maintenance. Design, setup and ...
SR Java Developer Team Location Cleveland Ohio
SR. Java Developer Team
Job Title : Senior Java Developer Team Lead
Job Location : Cleveland, OH
Job Description : Recruiting for a Senior Java Developer Team Lead to join the Information Services Division at our world headquarters in Cleveland, Ohio.
Will lead a team of ...
Software Engineering Manager Job Location : Farmington Hills, MI
Software Engineering Manager
Job Location : Farmington Hills, MI
Job Description : Job Responsibilities:
Lead a team of direct and indirect global software engineers in developing and verifying software for Driver Assist Group.•Provide technical and project leadership for ...
Catia V6 Developer (1971-EC)
We are searching for a Catia V6 developer to join our client's team in
Troy, MI.
You will be joining a small, nimble company that has grown to such an
extent that another Catia developer is needed. Ideally, you will be an
experienced Catia V6 developer, however a strong V5 ...
Dec 21 -
SAP Basis Team Lead location Cleveland Ohio Urgent need
SAP Basis Team Lead
Job Location : Cleveland, OH
Job Description : This position is responsible for leading a team of Basis administrators to effectively deliver the ongoing maintenance while also supporting the application teams with ...
Senior JavaScript Engineer (1969-EC)
Senior JavaScript Engineer needed to join a cutting edge Web and
mobile application development team.
You will join a team of smart, capable, passionate front-end
developers working on sophisticated Web applications supporting
Firefox, Chrome, Safari and IE. You will also ...
Dec 20 -
Ann Arbor
Systems Administrator
Systems Administrator, Farmington Hills, Michigan Kelly Services is currently seeking a Systems Administrator for one of our top clients in Farmington Hills, Michigan. As a Systems Administrator placed with Kelly Services, you will be responsible for installing, ...
Dec 19 -
Farmington Hills
Seeking ArcSight Consultants - Hiring Nationally!
Hiring in all Continental United States! Top Pay! Direct Hire!
IT Security SIEM Consultants
Travel / Work Remote from Home!
ArcSight Consultant are responsible for pre and post-sales product and service implementation, completing assigned projects in a manner ...
Dec 19 -
Hardware Monitor, Printer Technician Needed in Ann Arbor, MI, USA
Vodastra Solutions is the fastest growing preferred full service IT Solutions firm. Dedicated to connecting Top-notch quality technicians to multiple facilities needing IT services and, Solutions. We provide IT support in the U.S.A. and, Canada, specializing in Technical ...
Dec 19 -
Ann Arbor
Junior Software Developer
The Company
Go beyond. At Towers Watson, we are more than individuals. We are a community focused on helping our colleagues and clients thrive and succeed. Towers Watson is a leading global professional services company that helps organizations improve performance ...


text:
Click here to print the obituary below
Richard "Dick" Grosskopf
Grosskopf, Richard F. "Dick" 86, a 47 year resident of Wadsworth, was born April 6, 1927 in Pella, WI to the late Milton and Sylvia Grosskopf and passed away October 10, 2013 at the Alliance Hospice House in Pleasant Prairie, WI, surrounded by his wife and family. Dick was the beloved husband of Mary E. (nee Carlson), whom he met in Janesville, WI. They were married in 1954 and celebrated 59 years of marriage. Dick was the loving father of Jeffrey (Beth), Thomas (Jo Ann), Julie, David and Richard F. Jr. (Lori) Grosskopf; cherished grandfather of Melissa, Elyse, Lauren, Bryn, Brad, Eric, Jessica and Danielle and fond brother of the late Janice Ripley. He graduated from Shawano Community High School in 1945. While there he was an accomplished student athlete earning 11 varsity letters in 4 sports (football, basketball, golf and tennis) and appearing in Wisconsin State tournament finals for tennis and golf. He was inducted into the Shawano Community High School Sports Hall of Fame in 2003. Immediately upon high school graduation in the spring of 1945, Dick enlisted in the
United States Navy
and was stationed at Great Lakes Naval Base. After his military service, Dick returned to his home state and attended the
University of Wisconsin
-Madison where he earned a B.A. in Business Administration in 1950. He was a member of the Chi Phi Fraternity while at the University. After college he began a career as an insurance adjuster for the General Adjustment Bureau which led him to a prosperous career as a public insurance adjuster and insurance restoration contractor in Lake County for over 30 years. He was a past Waukegan City Club member and parish family member at St. Patrick Church in Wadsworth. Dick was an avid sportsman, involved in golf, hunting, and fishing. He held a family membership at Midlane Country Club. He captained the Serenity out of Waukegan Harbor and created many a fisherman's tale for those that came aboard. Visitation will be Monday, October 14, 2013 from 4:008:00 p.m. at Marsh Funeral Home 305 N. Cemetery Road, Gurnee. Family and friends will gather at 10:45 a.m. Tuesday, October 15 at St. Patrick Church 15000 Wadsworth Road, Wadsworth for funeral mass at 11:00 a.m. Interment will be private. Contributions in Dick's name may be made to Hospice Alliance, Inc., 10220 Prairie Ridge Boulevard, Pleasant Prairie, WI 53158. Funeral info 847-336-0127 or
Funeral Home
Marsh Funeral Home
305 N Cemetery Rd
Gurnee, IL 60031
(847) 336-0127
Published in News Sun from Oct. 12 to Oct. 18, 2013


text:
*TWINKLES (2-3 y/o):
The Twinkles are our first introduction to ballet and international dance for little students 2 to 3 years of age. Students repeat the basic ballet steps and also learn folk steps from countries around the world based on the repertoire of Presidio Dance Theatre.
*PRINCESS 101 (3-5 y/o):
PDT's Princess 101 classes are very popular, as they incorporate dance, music, dance history through storytelling, stage arts, and arts/crafts projects. Each week the students learn about a Ballet Princess (from the great ballets) and her court. Parents have raved about the team approach to teaching this class with arts specialists. Last semester, Moms put together these classes for their daughter's friends. It worked out very well and we do recommend that you consider doing this this for a small group of your princesses at a time/date that is most convenient for you.
*The Shirleys (3-5 y/o):
We just love old Shirley Temple movies. This class is be based on the dance choreography, music, and acting/pantomime from those movies. Students will require tap shoes for this class. This is a fun way to introduce young children to dance, singing, and acting. The Shirleys will perform in our school recitals.


text:
"One hundred and sixty eight ( 68 men and 10 0 women) undergraduates from a small, private college in Pennsylvania participate d in this study."
(L. McDermott, T. Pettijohn II: "The Influence of Clothing Fashion and Race on the Perceived Socioeconomic Status and Person Perception of College Students." Psychology & Society, 2011, Vol. 4 (2) , 64 ‐ 75
Draper: "What do women want?"
Stirling: "Who cares!"
One of my colleagues at Max-Planck-Institut once came to me with a draft paper. It dealt with dimorphism in sexes and would present evidence, that most differences could be explained from genetic heritage. The method that was mandatory practice at this institute was social biology -every behavior should only occur with humans and animals likewise, if a clear evolutionary advantage could be derived from it. Since it was the early 90s, the fight of science against postmodernism was still at its peak. Postmodernist thinking, like "it could just be our imposing social conventions into our methods to learn what we already knew", was brusquely brushed away, because "we use the scientific method, don't we?".
In the paper, my colleague presented results of some surveys he had conducted that showed correlation of the perceived "beauty" of people on images with the (I forgot how he had quantified it) beauty of the subjects spouse (I also forgot the thesis he would derive thereof). The correlation was very weak, like r²~0.6 or so. But because he had surveyed several hundred people, it became significant; it proved his absurd postulate.
My example might sound bizarre for the layperson, but have a look in publications like Journal for Personality and Social Psychology (where my colleague had issued his paper to).
The scientific method in general, but in quantitative social sciences in particular, has four steps to take:
1. Formulate the hypothesis
2. Draw a representative sample of observations
3. Test the hypothesis and prove it significant
4. Publish the results for review.
For now I do not want to focus on the strange reviewing practices that do not really publish results but rather keep them within closely confined boundaries of scientific journals, inaccessible for the public, only available for a small academic elite, so that a sound review hardly takes place.
I want to discuss the first three steps, because during the last 20 years, my professional field has undergone a dramatic paradigm shift, regarding these, while the forth is still holding for the time being.
The quantitative methods in social science originate from the age of the national mass society. They were developed as tools to help management and politics with their decisions. Alternatives to be tested where usually simple. The industrial production process would not allow for subtile variations in the product, thus it would be sufficient to present very few – usually two – varieties to the survey's subjects. People's lives would be likewise simple: a teacher's wife would show a distinctive consumption pattern, as would a coal miner. It would be good enough to know people's age, gender, and profession to generalize from one specimen to the whole group. Representativeness means, that one element of a set is used to represent the whole set – and not just with the properties, that would characterize the set itself (like male/female, Caucasian/Asian, etc.); the whole set would inherit all properties of its representative. It is counting the set as one, as Alain Badiou puts it.
We are so used to this aggregation of people into homogeneous sets, that we hardly realize its existence anymore. The concept of "target groups" in advertising is justified with this, too. Brands buy advertising by briefing the agency with gender, age, education of the people the campaign should reach. Prominent is the ABC-Audience in the UK, a rough segmentation of the populace just by their buying power and cultural capital.
In the mass society, up to the 1970s, this more or less seamed made sense. People in their class or milieu would behave sufficiently predictable. Especially television consumption still mirrors this aspect of mass society: ratings and advertising effect could be calculated and even predicted from the TV measurement panels with scientific precision. 2006 I came in charge of managing one of the largest and longest running social surveys, the "Typologie der Wünsche". Topics covered where consumption, brand preferences, and many aspects of people's opinions and daily routine, surveyed by personal interview of 10,000 participants per year. Preparing a joint study with Roland Berger Strategy Consultants, I examined the buyers of car brands regarding all aspects of them being defined as a "target group". The fascinating result was: while in the 1980s to the mid 1990s, buyers of car brands had been indeed quite homogeneous regarding their political opinions, ecological preferences, consumption of other brands, etc., this seemed to wane away over the last decade. The variance increased dramatically, so that to speak of "the buyer of a car brand" could be questioned. This was even more true for fast-moving consumer goods. Superficially, this could be explained by daily consumption becoming cheaper in proportion to average income, so poorer consumers where no longer as much restricted to certain goods as in earlier times. But the observation would prevail, even if just people with comparable wealth were taken into account.
Our conclusion: the end of mass media (that my employer was suffering from, like most traditional publishers), might come along with the end of mass society, too. The concept of aggretating people by objective criteria, by properties observable from the outside, like gender, income, or education, was getting under pressure.
For Israeli military strategist Martin van Creveld, this is also the underlying condition for what he calls the "Transfromation of War". In military philosophy, the corresponding paradigm is the idea of soldiers and civilians, unfolded by von Clausewitz. Van Crefeld argues, that the constructs of Clausewitz' theory, like 'peoples' ('Völker') had never existed in the first place. They were just stories told to organize war at industrial scale. And van Crefeld explicitly deconstructs the gender gap in battle. His book is full of quantitative proof that men, just regarded from the physical perspective, would make no better soldiers, than women. The the distributions of women and men in size, weight, physical strength etc. are mainly overlapping. Of course, the mean size of men is taller for a few centimeters than the size of women. This mean difference is significant, if you make a t-test. But as always, a significant mean difference says nothing about the individual. Most women are as tall as most men. Just some men are taller, and some women are smaller.
The fallacy of significance-testing should be obvious. It presumes the subjects would be originating from different universes, disjoint subsets of the population. Testing takes it for granted, that hypothesis and alternative are truly distinct, that only one can hold. This is hardly ever the case when humans are concerned. For most properties that we are studying in social research, the intra-set variance is much bigger than the variance between two sets, be it gender, be it age, education, hair color, or what ever criteria we choose to form the subsets. Women in most aspects are in average less similar one to each other, than the value of the mean difference of women and men.
This given, the logical next question would be, if the method had been correct, at all. The conditions of the industrial age made it only possible to serve their products to aggregates of people. Representative democracy also only gives the choice between a handful party programmes. And mass media could in principle not match individual preferences. So it seamed logic to regard people in categories, too, without bothering that dichotomous variables like sex might not be appropriate to map people's gender. Quantitative social research just reproduced the ideological restrictions of mass society.
With the Web, people suddenly had the choice, not only regarding media, but also regarding consumption. And -surprise!- people do act individually, and the actions are so random, that no correlation holds for more than a couple of weeks. "Multi-optional consumer" is a helpless way to express, that the silos of segmentation no longer make sense. Of course nobody has ever encountered a multi-optional person; on the individual level, people's behavior is mostly continuous and perfectly consistent; it is just no longer about "what women want".
The Web also presented for the first time a tool to collect data describing (nearly) everyone on the individual level. However, with trillions of data points on billions of users, every difference between subgroups becomes significant, anyway. Dealing with results like in the example of my colleague's ethological study mentioned above, the problem comes from taking significance as absolute. No matter how small an effect is, as long as it is significant, it will be considered proven. But statistical inference was designed to suite sample sizes of some ten to some low in the thousands people. It is ill-suited to deal with big data.
The jokes about silly correlations with Google trends are thus totally correct. And this demonstrates also another aspect: significance and hypotheses testing is regarded as static while data remains dynamic. While at some point in time, a correlation of Google trends with other time series might just randomly become significant, it is highly unlikely, that this bogus correlation will survive. Data science, other than classic quantitative research, tends to deal with data in an agile way, which means, that nothing is regarded as fixed. But if we see our data as ephemeral, there is no need to come up with models that we restrict according to fixed proven hypothesis.
So the role of statistics for social science changes. Statistics is now the tool to deal with distributions as phenomena as such rather than just generalizing from small samples to an unknown population. We should use the stream data as the life-condition, our models would have to struggle to survive in. Like with biological evolution, we would not expect the assumptions to remain stable. We would rather expect, the boundary conditions to change, and our models would have to adjust; survival of the fittest model means: the fittest for now.
The philosophical justification for inference is the idea of the general comprehensibility of reality. Like St. Augustine we postulate that it is possible to extrapolate from perception (=measurement, data) to the world of things. But like our sensory organs have been evolving, driven by environmental change (and mutations in our genome), we should regard the knowledge we derive from data as "shadow on the cave wall" at best.
This is far better than it sounds: it gives us freedom to explore data rather than just test our made-up hypotheses, that would just perpetuate our presumptions.
Let's leave statistical testing and significance where it belongs to: Quality assurance, material testing, physical measurements -engineering.
Let's be honest, and drop it in the humanities.
Some more texts:
How to create a location graph from the Foursquare API
Monday, I'll be speaking on "Linked Data" at the 49th German Market Research Congress 2014. In my talk, there will be many examples of how to apply the basic approach and measurements of Social Network Analysis to various topics ranging from brand affinities as measured in the market-media study best for planning, the financial network between venture capital firms and start-ups and the location graph on Foursquare.
Because I haven't seen many examples on using the Foursquare API to generate location graphs, I would like to explain my approach a little bit deeper. At first sight, the Foursquare API differs from many other Social Media APIs because it just allows you to access data about your own account. So, there is no general stream (or firehose) of check-in events that could be used to calculate user journeys or the relations between different places.
Fortunately, there's another method that is very helpful for this purpose: You can query the API for any given Foursquare location to output up to five venues that were most frequently accessed after this location. This begs for a recursive approach of downloading the next locations for the next locations for the next locations and so on … and transform this data into the location graph.
I've written down this approach in an IPython Notebook, so you just have to find your API credentials and then you can start downloading your cities' location graph. For Munich it looks like this (click to zoom):
The resulting network is very interesting, because the "distance" between the different locations is a fascinating mixture of
- spatial distance: places that are nearby are more likely to be connected (think of neighborhoods)
- temporal distance: places that can be reached in a short time are more likely to be connected (think of places that are quite far apart but can be reached in no time by highway)
- affective/social distance: places that belong to a common lifestyle are more likely to be connected
Feel free to clone the code from my github. I'm looking forward to seeing the network visualizations of your cities.
"The NSA is basically applied data science."
Jason Cust
The European Court of Justice declared the EU directive on data retention void the same day #hartbleed caused the grosset panic about password security the Net had seen so far.
This tells one story:
Get out into the open! Stop hiding! There are no remote places. No privileges will keep your data from the public. No genius open source hack will protect your informational self determination. And spooks, listen: your voyerism is not accepted as OK, not even by the law.
We're living in a world with more transparency, we need to learn to do intelligence with more transparency too.
Bruce Schneier, #unowned
I was fighting against Europe's data retention directive, too, and of course I was feeling victorious when it was declared void by the ECJ. But, don't we know how futile all efforts are, to keep data protected? No cypher can be unbreakable, except the one-time-pad (and that is just of academic interest). No program-code can be proven safe, either – these are mathematical facts. Haven't you learned how wrong all promisses of security are by #heartbleed? Data protection lulls us into feeling save from data breeches, where we should rather care to make things robust, no matter if the data becomes public intentionally or not.
The far more important battle was won the same week, when the European Parliament decided to protect net neutrality by law. This is the real data protection for me: protecting the means of data production from being engrossed in private.
The intelligence agencies' damaging the Net by undermining the trust of its users in the integrity of its technology, is a serious thing by inself. However burrying this scandal just under the clutter of civil rights or under constitutional law's aspects will not do justice. It is of our living in societies in transition from the modern nation state to the after-modern … what ever, liquid community; into McLuhan's Global Village which is much more serious and much more interesting. We will have to go through the changes of the world and it might not be nice all the time until we come out the other end.
Data begs to be used.
Bruce Schneier, #unowned
In "Snow Crash", Neal Stephenson imagines a Central Intelligence Corporation, the CIA and NSA becoming a commercial service where everybody just purchases the information they'd need. This was, what came immediately to my mind when I read through the transcript of the talk on "Intelligence Gathering and the Unowned Internet" that was held by the Berkman Center for Internet and Society at Harvard, starring Bruce Schneier, who for every Mathematician in my generation is just the godfather of cryptography. Representatives of the intelligence community were also present. Bruce has been arguing for living "beyond fear" for more than a decade, advocating openness instead of digging trenches and winding up barbed wire. I am convinced, that information does not want to be free (as many of my comrades in arms tend to phrase). However I strongly belief Bruce is right: We can hear data's call.
The Quantified Self, life-logging, self-tracking – many people making public even their most intimate data, like health, mood, or even our visual field of sight at any moment of our day. An increasingly strong urge to do this rises from social responsibility – health care, climate research, but also from quite profane uses like insurances offering discounts when you let your driving be tracked. So the data is there now, about everything there is to know. Not using it because of fear for privacy would be like having abandond the steam engine and abstinate from the industrialization in anticipation of the climate change.
Speak with us, don't speak for us.
statement of autonomy of #OccupyWallStreet
Dinosaurs of modern warfare, like the NSA are already escamotating into dragons, fighting their death-match like Smaug at Lake Esgaroth. We have to deal with the dragon, kill it, but this is not, what we should set our political goal to. It will happen anyway.
We should take provision for what real changes we are going to face. We will find ourselves in a form of society that McLuhan would have called re-tribalized, or Tönnies would not have called society at all, but community. In a community, the concept of privacy is usually rather weak. But at the same time, there is no surveillance, no panoptic elevated watchmen who themself cannot be watched. Everyone is every other one's keeper. Communal vigilance doesn't sound like fun. However it will be the consequence of after-modern communal structures replacing the modern society. "In the electric world, we wear all mankind as our skin.", so McLuhan speaketh.
Noi vogliamo cantare l'amor del pericolo.
Every aspect of life gets quantized, datarized, tangible by computers. Our aggregated models of human behavior get replaced by exact description of the single person's life and thus the predictions of that person's future actions. We witness the rise of non-representative democratic forms like occupy assemblies or liquid democracy. What room can privacy have in a world like this whatsoever? So maybe we should concentrate in shaping our data-laden future, rather then protecting a fantasy of data being contained in some "virtual reality" that could be kept separate from our lives.
The data calls us from the depth; let us hear to its voice!
Further reading:
Transcript of the discussion "The unowned Internet"
Declaration of Liquid Culture
Disrupt Politics!
Analyzing VC investment strategies with Crunchbase data
If you look at the investments in Big Data companies in the last few years, one thing is obvious: This is a very dynamic and fast growing market. I am producing regular updates of this network map of Big Data investments with a Python program (actually an IPython Notebook).
But what insights can be gained by directly analyzing the Crunchbase invesmtent data? Today I revved up my RStudio to take a clooser look at the data beneath the nodes and links.
Load the data and required packages:
library(reshape) library(plyr) library(ggplot2) data <- read.csv('crunchbase_monthly_export_201403_investments.csv', sep=';', stringsAsFactors=F) inv <- data[,c("investor_name", "company_name", "company_category_code", "raised_amount_usd", "investor_category_code")] inv$raised_amount_usd[is.na(inv$raised_amount_usd)] <- 1
In the next step, we are selecting only the 100 top VC firms for our analysis:
inv <- inv[inv$investor_category_code %in% c("finance", ""),] top <- ddply(inv, .(investor_name), summarize, sum(raised_amount_usd)) names(top) <- c("investor_name", "usd") top <- top[order(top$usd, decreasing=T),][1:100,] invtop <- inv[inv$investor_name %in% top$investor_name[1:100],]
Right now, each investment from a VC firm to a Big Data company is one row. But to analyze the similarities between the VC companies in term of their investment in the various markets, we have to transform the data into a matrix. Fortunately, this is exactly, what Hadley Wickham's reshape package can do for us:
inv.mat <- cast(invtop[,1:4], investor_name~company_category_code, sum) inv.names <- inv.mat$investor_name inv.mat <- inv.mat[,3:40] # drop the name column and the V1 column (unknown market)
These are the most important market segments in the Crunchbase (Top 100 VCs only):
inv.seg <- ddply(invtop, .(company_category_code), summarize, sum(raised_amount_usd)) names(inv.seg) <- c("Market", "USD") inv.seg <- inv.seg[inv.seg$Market != "",] inv.seg$Market <- as.factor(inv.seg$Market) inv.seg$Market <- reorder(inv.seg$Market, inv.seg$USD) ggplot(inv.seg, aes(Market, USD/1000000))+geom_bar(stat="identity")+coord_flip()+ylab("$1M USD")
What's interesting now: Which branches are related to each other in terms of investments (e.g. VCs who invested in biotech also invested in cleantech and health …). This question can be answered by running the data through a K-means cluster analysis. In order to downplay the absolute differences between the categories, I am using the log values of the investments:
inv.market <- log(t(inv.mat)) inv.market[inv.market == -Inf] <- 0 fit <- kmeans(inv.market, 7, nstart=50) pca <- prcomp(inv.market) pca <- as.matrix(pca$x) plot(pca[,2], pca[,1], type="n", xlab="Principal Component 1", ylab="Principal Component 2", main="Market Segments") text(pca[,2], pca[,1], labels = names(inv.mat), cex=.7, col=fit$cluster)
My 7 cluster solution has identified the following clusters:
- Health
- Cleantech / Semiconductors
- Manufacturing
- News, Search and Messaging
- Social, Finance, Analytics, Advertising
- Automotive & Sports
- Entertainment
The same can of course be done for the investment firms. Here the question will be: Which clusters of investment strategies can be identified? The first variant has been calculated with the log values from above:
inv.log <- log(inv.mat) inv.log[inv.log == -Inf] <- 0 inv.rel <- scale(inv.mat) fit <- kmeans(inv.log, 6, nstart=15) pca <- prcomp(inv.log) pca <- as.matrix(pca$x) plot(pca[,2], pca[,1], type="n", xlab="Principal Component 1", ylab="Principal Component 2", main="VC firms") text(pca[,2], pca[,1], labels = inv.names, cex=.7, col=fit$cluster)
The second variant uses scaled values:
inv.rel <- scale(inv.mat) fit <- kmeans(inv.rel, 6, nstart=15) pca <- prcomp(inv.rel) pca <- as.matrix(pca$x) plot(pca[,2], pca[,1], type="n", xlab="Principal Component 1", ylab="Principal Component 2", main="VC firms") text(pca[,2], pca[,1], labels = inv.names, cex=.7, col=fit$cluster)
Datalicious Notebookmania – My favorite 7 IPython Notebooks
One of the most remarkable features of this year's Strataconf was the almost universal use of IPython notebooks in presentations and tutorials. This framework not only allows the speakers to demonstrate each step in the data science approach but also gives the audience an opportunity to do the same – either during the session or afterwards.
Here's a list of my favorite IPython notebooks on machine learning and data science. You can always find a lot more on this webpage. Furthermore, there's also the great notebookviewer platform that can render Github'bed notebooks as they would appear in your browser. All the following notebooks can be downloaded or cloned from the GitHub page to work on your own computer or you can view (but not edit) them with nbviewer.
So, if you want to learn about predictions, modelling and large-scale data analysis, the following resources should give you a fantastic deep dive into these topics:
1) Mining the Social Web by Matthew A. Russell
If you want to learn how to automatically extract information from Twitter streams, Facebook fanpages, Google+ posts, Github accounts and many more information sources, this is the best resource to start. It started out as the code repository for Matthew's O'Reilly published book, but since the 2nd edition has become an active learning community. The code comes with a complete setup for a virtual machine (Vagrant based) which saves you a lot of configuring and version-checking Python packages. Highly recommended!
2) Probabilistic Programming and Bayesian Methods for Hackers by Cameron Davidson-Pilon
This is another heavy weight among my IPython notebook repositories. Here, Cameron teaches you Bayesian data analysis from your first calculation of posteriors to a real-time analysis of GitHub repositories forks. Probabilistic programming is one of the hottest topics in the data science community right now – Beau Cronin gave a mind-blowing talk at this year's Strata Conference (here's the speaker deck) – so if you want to join the Bayesian gang and learn probabilistic programming systems such as PyMC, this is your notebook.
3) Parallel Machine Learning Tutorial by Olivier Grisel
The tutorial session on parallel machine learning and the Python package scikit-learn by Olivier Grisel was one of my highlights at Strata 2014. In this notebook, Olivier explains how to set up and tune machine learning projects such as predictive modeling with the famous Titanic data-set on Kaggle. Modeling has far too long been a secret science – some kind of Statistical Alchemy, see the talk I gave at Siemens on this topic – and the time has come to democratize the methods and approaches that are behind many modern technologies from behavioral targeting to movie recommendations. After the introduction, Olivier also explains how to use parallel processing for machine learning projects on really large data-sets.
4) 538 Election Forecasting Model by Skipper Seabold
Ever wondered how Nate Silver calculated his 2012 presidential election forecasts? Don't look any further. This notebook is reverse engineering Nate's approach as he described it on his blog and in various interviews. The notebook comes with the actual polling data, so you can "do the Nate Silver" on your own laptop. I am currently working on transforming this model to work with German elections – so if you have any ideas on how to improve or complete the approach, I'd love to hear from you in the comments section.
5) Six Degrees of Kevin Bacon by Brian Kent
This notebook is one of the showcases for the new GraphLab Python package demonstrated at Strata Conference 2014. The GraphLab library allows very fast access to large data structures with a special data frame format called the SFrame. This notebook works on the Freebase movie database to find out whether the Kevin Bacon number really holds true or whether there are other actors that are more central in the movie universe. The GraphLab package is currently in public beta.
6) Get Close to Your Data with Python and JavaScript by Brian Granger
The days of holecount and 1000+ pages of statistical tables are finally history. Today, data science and data visualization go together like Bayesian priors and posteriors. One of the hippest and most powerful technologies in modern browser-based visualization is the d3.js framework. If you want to learn about the current state-of-the-art in combining the beauty of d3.js with the ease and convenience of IPython, Brian's Strata talk is the perfect introduction to this topic.
I found the final notebook through the above mentioned talk. Peter Norvig is not only the master mind behind the Google economy, teacher of a wonderful introduction to Python programming at Udacity and author of many scientific papers on applied statistics and modeling, but he also seems to be the true nerd. Who else would take a xkcd comic strip by the word and work out the regular expression matching patterns that provide a solution to the problem posed in the comic strip. I promise that your life will never be the same after you went through this notebook – you'll start to see programming problems in almost every Internet meme from now on. Let me know, when you found some interesting solutions!
Statistics is often regarded as the mathematics of gambling, and it has some roots in theorizing about games, indeed. But it was the steam engine that really made statistics do something: Thermodynamics, the physics of heat, energy, and gases. Aggregating over huge masses of particles – not observable on an individual level – by means of probability distribution was the paradigm of 19th century science. And this metaphor also was successfully adopted to describing not only masses of molecules, but also masses of people in a mass society.For physics, at the end of the 19th century it had become clear, that models reduced on aggregates and distributions where not able to explain many observations that where experimentally proven, like black body radiation or the photo-electric effect. It was Max Planck and Albert Einstein that moved the perspective from statistical aggregates to something that had not been usually taken into consideration: the particle. Quantum physics is the description of physical phenomena on the most granular level possible. By changing focus from the indistinct mass to the individual particle, also the macroscopic level of physics started to make sense again, combining probabilistic concepts like entropy with the behavior of the single particle that we might visualize in a Feynman-diagram. The Web presented for the first time a tool to collect data describing (nearly) everyone on the individual level. The best data came not from intentional research but from cookie-tracking, done to optimize advertising effectiveness. Social Media brought us the next level: semantic data, people talking about their lives, their preferences, their actions and feelings. And people connected with each other, the social graph showed who was talking to whom and about which topics – and how tight social bonds were knit.
We now have the data to model behavior without the need of aggregating. The role of statistics for the humanities changes – like it has done in physics 150 years ago. Statistics is now the tool to deal with distributions as phenomena as such rather than just generalizing from small samples to an unknown population. 'Data humanity' would be a much better term for what is usually called 'data science' – this I had written after O'Reilly's Strata conference last year. But I think I might have been wrong as we move from social science to computational social science.
Social research is moving from humanities to science.
Further reading:
Open foresight is a great way to look into future developments. Open data is the foundation to do this comprehensively and in a transparent way. As with most big data projects, the difficult part in open foresight is to collect the data and wrangle it to a form that can actually be processed. While in classic social research you'd have experimental measurements or field notes in a well defined format, dealing with open data is always a pain: not only is there no standard – the meaningful numbers might be found anywhere in your source and be called arbitrarily; also the context is not given by some structure that you'd have imposed into your data in advanced (as we used to do it in our hypothesis-driven set-ups).
In the last decade, crowdsourcing has proven to be a remedy to dealing with all kinds of challenges that are still to complex to be fully automatized, but which are not too hard to be worked out by humans. A nice example is zooniverse.org featuring many "citizen science projects", from finding exoplanets or classifying galaxies, to helping to model global climate history by entering historic ships' log data.
Climate change caused by humanity might be the best defended hypothesis in science; no other theory had do be defended against more money and effort to disprove it (except perhaps evolution, which has do fight a similar battle about ideology). But apart from the description, how climate will change and how that will effect local weather conditions, we might still be rather little aware of the consequences of different scenarios. But aside from the effect of climate-driven economic change on people's lives, the change of economy itself cannot be ignored when studying climate and understand possible feedback loops that might or might not lead into local or global catastrophe.
Zeean.net is an open data / open source project aiming at the economic impact of climate change. Collecting data is crowdsourced – everyone can contribute key indicators of geo-economic dependency like interregional and domestic flow of supply and demand in an easy "Wikipedia-like" way. And like Wikipedia, the validation is done by crowd-crosscheck of registered users. Once data is there, it can be fed into simulations. The team behind Zeean, lead by Anders Levermann at Potsdam Institute for Climate Impact Research is directly tied into the Intergovernmental Panel on Climate Change IPCC, leading research on climate change for the UN and thus being one of the most prominent scientific organizations in this field.The results are of course not limited to climate. If markets default for other reasons, the effect on other regions can be modeled in the same way.
So I am looking forward to the data itself being made public (by then brought into a meaningful structure), we could start calculating our own models and predictions, using the powerful open source tools that have been made available during the last years.
Before and After Series C funding – a network analysis of Domo
One of the most interesting Big Data companies in this network analysis of Venture Capital connections has in my opinion been Domo. Not only did it receive clearly above average funding for such a young company, but it was also one of the nodes with the best connections through Venture Capital firms and their investments. It had one of the highest values for Betweenness Centrality, which means it connects a lot of the other nodes in the Big Data landscape.
Then, some days after I did the analysis and visualization, news broke that Domo received $125M from Greylock, Fidelity, Morgan Stanley and Salesforce among others. This is a great opportunity to see what this new financing round means in terms of network structure. Here's Domo before the round:
And this is Domo $125M later. Notice how its huge Betweenness Centrality almost dwarfs the other nodes in the network. And through its new connections it is strongly connected to MongoDB:
Here's a look at the numbers, before Series C:
|The Climate Corporation
|MongoDB, Inc.
|Greenplum Software
And now:
|MongoDB, Inc.
|The Climate Corporation
|Greenplum Software
The new funding round now only increases Domo's centrality but also MongoDB's because of the shared investors Salesforce, T. Rowe Price and Fidelity Investments.
As the data-base for the Big Data Investment Map 2014 also includes the dates for most of the funding rounds, it's not hard to create a time-series plot from this data. This should answer the question whether Big Data is already over the peak (cf. Gartner seeing Big Data reaching the "trough of disillusionment") or if we still are to experience unseen heights? The answer should be quite clear:
The growth does look quite exponential to me. BTW: The early spike in 2007 has been the huge investment in VMWare by Intel and Cisco. Currently, I have not included IPOs and acquisitions in my calculations.
Here's an updated version of our Big Data Investment Map. I've collected information about ca. 50 of the most important Big Data startups via the Crunchbase API. The funding rounds were used to create a weighted directed network with investments being the edges between the nodes (investors and/or startups). If there were multiple companies or persons participating in a funding round, I split the sum between all investors.
This is an excerpt from the resulting network map – made with Gephi. Click to view or download the full graphic:
If you feel, your company is missing in the network map, please tell us in the comments.
The size of the nodes is relative to the logarithmic total result of all their funding rounds. There's also an alternative view focused on the funding companies – here, the node size is relative to their Big Data investments. Here's the list of the top Big Data companies:
(M$, Source: Crunchbase API)
|Palantir Technologies
|MongoDB, Inc.
|The Climate Corporation
And here's the top investing companies:
|Big Data funding
(M$, Source: Crunchbase API)
|Founders Fund
|New Enterprise Associates
|Sequoia Capital
|General Electric
|Accel Partners
|Lightspeed Venture Partners
|Greylock Partners
|Meritech Capital Partners
We can also use network analytical measures to find out about which investment company is best connected to the Big Data start-up ecosystem. I've calculated the Betweenness Centrality measure which captures how good nodes are at connecting all the other nodes. So here are the best connected Big Data investors and their investments starting with New Enterprise Associates, Andreessen Horowitz and In-Q-Tel (the venture capital firm for the CIA and the US intelligence community).
|Big Data Companies
|New Enterprise Associates
|GraphLab, MapR Technologies, Fusion-io, MongoDB, Inc., WibiData, Pentaho, CloudFlare, The Climate Corporation, MemSQL
|Andreessen Horowitz
|ClearStory Data, Domo, Fusion-io, Databricks, GoodData, Continuuity, Platfora
|Cloudera, Recorded Future, Cloudant, MongoDB, Inc., Platfora, Palantir Technologies
|Founders Fund
|Declara, CrowdFlower, The Climate Corporation, Palantir Technologies, Domo
|SV Angel
|Cloudera, Domo, WibiData, Citus Data, The Climate Corporation, MemSQL
|Khosla Ventures
|ParStream, Metamarkets, MemSQL, ClearStory Data, The Climate Corporation
|IA Ventures
|Metamarkets, Recorded Future, DataSift, MemSQL
|Data Collective
|Trifacta, ParStream, Continuuity, Declara, Citus Data, Platfora, MemSQL
|Hummer Winblad Venture Partners
|NuoDB, Karmasphere, Domo
|Battery Ventures
|Kontagent, SiSense, Continuuity, Platfora
Once a year, the cosmopolitan digital avantgarde gathers in Munich to listen to keynotes on topics all the way from underground gardening to digital publishing at the DLD, hosted by Hubert Burda. In the last years, I did look at the event from a network analytical perspective. This year, I am analyzing the content, people were posting on Twitter in order to make comparisons to last years' events and the most important trends right now.
To do this in the spirit of Street Fighting Trend Research, I limited myself to openly available free tools to do the analysis. The data-gathering part was done in the Google Drive platform with the help of Martin Hawksey's wonderful TAGS script that is collecting all the tweets (or almost all) to a chosen hashtag or keyword such as "#DLD14″ or "#DLD" in this case. Of course, there can be minor outages in the access to the search API, that appear as zero lines in the data – but that's not different to data-collection e.g. in nanophysics and could be reframed as adding an extra challenge to the work of the data scientist The resulting timeline of Tweets during the 3 DLD days from Sunday to Tuesday looks like this:
You can clearly see three spikes for the conference days, the Monday spike being a bit higher than the first. Also, there is a slight decline during lunch time – so there doesn't seem to be a lot food tweeting at the conference. To produce this chart (in IPython Notebook) I transformed the Twitter data to TimeSeries objects and carefully de-duplicated the data. In the next step, I time shifted the 2013 data to find out how the buzz levels differed between last years' and this years' event (unfortunately, I only have data for the first two days of DLD 2013.
The similarity of the two curves is fascinating, isn't it? Although there still are minor differences: DLD14 began somewhat earlier, had a small spike at midnight (the blogger meeting perhaps) and the second day was somewhat busier than at DLD13. But still, not only the relative, but also the absolute numbers were almost identical.
Now, let's take a look at the devices used for sending Tweets from the event. Especially interesting is the relation between this years' and last years' percentages to see which devices are trending right now:
The message is clear: mobile clients are on the rise. Twitter for Android has almost doubled its share between 2013 and 2014, but Twitter for iPad and iPhone have also gained a lot of traction. The biggest losers is the regular Twitter web site dropping from 39 per cent of all Tweets to only 22 per cent.
The most important trending word is "DLD14″, but this is not surprising. But the other trending words allow deeper insights into the discussions at the DLD: This event was about a lot of money (Jimmy Wales billion dollar donation), Google, Munich and of course the mobile internet:
Compare this with the top words for DLD 2013:
Wait – "sex" among the 25 most important words at this conference? To find out what's behind this story, I analyzed the most frequently used bigrams or word combinations in 2013 and 2014:
With a little background knowledge, it clearly shows that 2013′s "sex" is originating from a DJ Patil quote comparing "Big Data" (the no. 1 bigram) with "Teenage Sex". You can also find this quotation appearing in Spanish fragments. Other bigrams that were defining the 2013 DLD were New York (Times) and (Arthur) Sulzberger, while in 2014 the buzz focused on Jimmy Wales, Rovio and the new Xenon processor and its implications for Moore's law. In both years, a significant number of Tweets are written in Spanish language.
UPDATE: Here's the IPython Notebook with all the code, this analysis has been based on.
Trending Topics at Strata Conferences 2011-2014
To fill the gap until this year's Strata Conference in Santa Clara, I thought of a way to find out trends in big data and data science. As this conference should easily be the leading edge gathering of practitioners, theorists and followers of big data analytics, the abstracts submitted and accepted for Strataconf should give some valuable input. So, I collected the abstracts from the last Santa Clara Strata conferences and applied some Python nltk magic to it – all in a single IPython Notebook, of course.
Here's a look at the resulting insights. First, I analyzed the most frequent words, people used in their abstracts (after excluding common English language stop words). As a starter, here's the Top 20 words for the last four Strata conferences:
This is just to check, whether all the important buzzwords are there and we're measuring the right things here: Data – check! Hadoop – check! Big – check! Business – check! Already with this simple frequency count, one thing seems very interesting: Hadoop didn't seem to be a big topic in the community until 2012. Another random conclusion could be that 2011 was the year where Big Data really was "new". This word loses traction in the following years.
And now for something a bit more sophisticated: Bigrams or frequently used word combinations:
Of course, the top bigram through all the years is "big data", which is not entirely unexpected. But you can clearly see some variation among the Top 20. Looking at the relative frequency of the mentions, you can see that the most important topic "Big Data" will probably not be as important in this years conference – the topical variety seems to be increasing:
Looking at some famous programming and mathematical languages, the strong dominance of R seems to be broken by Python or IPython (and its Notebook environment) which seems to have established itself as the ideal programming tool for the nerdy real-time presentation of data hacks. \o/
Another trend can be seen in the following chart: Big Data seems to become more and more faceted over the years. The dominant focus on business applications of data analysis seems to be over and the number of different topics discussed on the conference seems to be increasing:
Finally, let's take a more systematic look at rising topics at Strata Conferences. To find out which topics were gaining momentum, I calculated the relative frequencies of all the words and compared them to the year before. So, here's the trending topics:
These charts show that 2012 was indeed the "Hadoop-Strata" where this technology was the great story for the community, but also the programming language R became the favorite Swiss knife for data scientists. 2013 was about applications like Hive that run on top of Hadoop, data visualizations and Google seemed to generate a lot of buzz in the community. Also, 2013 was the year, data really became a science – this is the second most important trending topic. And this was exactly the way, I experienced the 2013 Strata "on the ground" in Santa Clara.
What will 2014 bring? The data suggests, it will be the return of the hardware (e.g. high performance clusters), but also about building data architectures, bringing data know-how into organizations and on a more technical dimension about graph processing. Sounds very promising in my ears!
Identifying trends in the German Google n-grams corpus (Tutorial)
A lot of people still have a lot of respect for Hadoop and MapReduce. I experience it regularly in workshops with market researchers and advertising people. Hadoop's image is quite comparable with Linux' perceived image in the 1990s: a tool for professional users that requires a lot of configuration. But in the same way, there were some user-friendly distributions (e.g. Suse), there are MapReduce tools that require almost no configuration.
One favorite example is the ease and speed, you can do serious analytical work on the Google n-grams corpus with Hive on Amazon's Elastic MapReduce platform. I adapted the very helpful code from the AWS tutorial on the English corpus to find out the trending German words (or 1-grams) for the last century. You need to have an Amazon AWS account and valid SSH keys to connect to the machines you are running the MapReduce programs on (here's the whole hive query file).
- Start your Elastic MapReduce cluster on the EMR console. I used 1 Master and 19 slave nodes. Select your AWS ssh authorization key. Remember: from this moment on, your cluster is generating costs. So, don't forget to terminate the cluster after the job is done!
- If your Cluster has been set-up and is running, note the Master-Node-DNS. Open a SSH client (e.g. Putty on Windows or ssh on Linux) and connect to the master node with the ssh key. Your username on the remote machine is "hadoop".
- Start "hive" and set some useful defaults for the analytical job:
set hive.base.inputformat=org.apache.hadoop.hive.ql.io.HiveInputFormat;
set mapred.min.split.size=134217728;
- The first code snippet connects to the 1-gram dataset which resides on the S3 storage:
CREATE EXTERNAL TABLE german_1grams (
gram string,
year int,
occurrences bigint,
pages bigint,
books bigint
ROW FORMAT DELIMITED FIELDS TERMINATED BY '\t'
LOCATION 's3://datasets.elasticmapreduce/ngrams/books/20090715/ger-all/1gram/';
- Now, we can use this database to perform some operations. The first step is to normalize the database, e.g. to transform all words to lower case and remove 1-grams that are no proper words. Of course you could further refine this step to remove stopwords or reduce the words to their stems by stemming or lemmatization.
CREATE TABLE normalized (
gram string,
year int,
occurrences bigint
And then we populate this table:
INSERT OVERWRITE TABLE normalized
year >= 1889 AND
gram REGEXP "^[A-Za-z+'-]+$";
- The previous steps should run quite fast. Here's the step that really need to be run on a multi-machine cluster:
CREATE TABLE by_decade (
gram string,
decade int,
ratio double
INSERT OVERWRITE TABLE by_decade
sum(a.occurrences) / b.total
normalized a
substr(year, 0, 3) as decade,
sum(occurrences) as total
substr(year, 0, 3)
) b
substr(a.year, 0, 3) = b.decade
- The final step is to count all the trending words and export the data:
CREATE TABLE result_decade (
gram string,
decade int,
ratio double,
increase double );
INSERT OVERWRITE TABLE result_decade
a.gram as gram,
a.decade as decade,
a.ratio as ratio,
a.ratio / b.ratio as increase
by_decade a
by_decade b
a.gram = b.gram and
a.decade - 1 = b.decade
a.ratio > 0.000001 and
a.decade >= 190
decade ASC,
increase DESC;
- The result is saved as a tab delimited plaintext data file. We just have to find out its correct location and then transfer it from the Hadoop HDFS file system to the "normal" file system on the remote machine and then transfer it to our local computer. The (successful) end of the hive job should look like this on your ssh console:
The line "Deleted hdfs://x.x.x.x:9000/mnt/hive_0110/warehouse/export" gives you the information where the file is located. You can transfer it with the following command:
$ hdfs dfs -cat /mnt/hive_0110/warehouse/export/* > ~/export_file.txt
- Now the data is in the home directory of the remote hadoop user in the file export_file.txt. With a secure file copy program such as scp or WinSCP you can download the file to your local machine. On a Linux machine, I should have converted the AWS SSH key in the Linux format (id_rsa and id_rsa.pub) and then added. With the following command I could download our results (replace x.x.x.x with your IP address or the Master-Host-DNS):
$ scp email@example.com:export_file.txt ~/export_file.txt
- After you verified that the file is intact, you can terminate your Elastic MapReduce instances.
As a result you get a large text file with information on the ngram, decade, relative frequency and growth ratio in comparison with the previous decade. After converting this file into a more readable Excel document with this Python program, it looks like this:
Values higher than 1 in the increase column means that this word has grown in importance while values lower than 1 means that this word had been used more frequently in the previous decade.
Here's the top 30 results per decade:
- 1900s: Adrenalin, Elektronentheorie, Textabb, Zysten, Weininger, drahtlosen, Mutterschutz, Plazenta, Tonerde, Windhuk, Perseveration, Karzinom, Elektrons, Leukozyten, Housz, Schecks, kber, Zentralwindung, Tarifvertrags, drahtlose, Straftaten, Anopheles, Trypanosomen, radioaktive, Tonschiefer, Achsenzylinder, Heynlin, Bastimento, Fritter, Straftat
- 1910s: Commerzdeputation, Bootkrieg, Diathermie, Feldgrauen, Sasonow, Wehrbeitrag, Bolschewismus, bolschewistischen, Porck, Kriegswirtschaft, Expressionismus, Bolschewiki, Wirtschaftskrieg, HSM, Strahlentherapie, Kriegsziele, Schizophrenie, Berufsberatung, Balkankrieg, Schizophrenen, Enver, Angestelltenversicherung, Strahlenbehandlung, Orczy, Narodna, EKG, Besenval, Flugzeugen, Flugzeuge, Wirkenseinheit
- 1920s: Reichsbahngesellschaft, Milld, Dawesplan, Kungtse, Fascismus, Eidetiker, Spannungsfunktion, Paneuropa, Krestinski, Orogen, Tschechoslovakischen, Weltwirtschaftskonferenz, RSFSR, Sachv, Inflationszeit, Komintern, UdSSR, RPF, Reparationszahlungen, Sachlieferungen, Konjunkturforschung, Schizothymen, Betriebswirtschaftslehre, Kriegsschuldfrage, Nachkriegsjahre, Mussorgski, Nachkriegsjahren, Nachkriegszeit, Notgemeinschaft, Erlik
- 1930s: Reichsarbeitsdienst, Wehrwirtschaft, Anerbengericht, Remilitarisierung, Steuergutscheine, Huguenau, Molotov, Volksfront, Hauptvereinigung, Reichsarbeitsdienstes, Viruses, Mandschukuo, Erzeugungsschlacht, Neutrons, MacHeath, Reichsautobahnen, Ciano, Vierjahresplan, Erbkranken, Schuschnigg, Reichsgruppe, Arbeitsfront, NSDAP, Tarifordnungen, Vierjahresplanes, Mutationsrate, Erbhof, GDI, Hitlerjugend, Gemeinnutz
- 1940s: KLV, Cibazol, UNRRA, Vollziehungsrath, Bhil, Verordening, Akha, Sulfamides, Ekiken, Wehrmachtbericht, Capsiden, Meau, Lewerenz, Wehrmachtsbericht, juedischen, Kriegsberichter, Rourden, Gauwirtschaftskammer, Kriegseinsatz, Bidault, Sartre, Riepp, Thailands, Oppanol, Jeftanovic, OEEC, Westzonen, Secretaris, pharmaceutiques, Lodsch
- 1950s: DDZ, Peniteat, ACTH, Bleist, Siebenjahrplan, Reaktoren, Cortison, Stalinallee, Betriebsparteiorganisation, Europaarmee, NPDP, SVN, Genossenschaftsbauern, Grundorganisationen, Sputnik, Wasserstoffwaffen, ADAP, BverfGg, Chruschtschows, Abung, CVP, Atomtod, Chruschtschow, Andagoya, LPG, OECE, LDPD, Hakoah, Cortisone, GrundG
- 1960s: Goldburg, Dubcek, Entwicklungszusammenarbeit, Industriepreisreform, Thant, Hoggan, Rhetikus, NPD, Globalstrategie, Notstandsgesetze, Nichtverbreitung, Kennedys, PPF, Pompidou, Nichtweiterverbreitung, neokolonialistischen, Teilhards, Notstandsverfassung, Biafra, Kiesingers, McNamara, Hochhuth, BMZ, OAU, Dutschke, Rusk, Neokolonialismus, Atomstreitmacht, Periodikums, MLF
- 1970s: Zsfassung, Eurokommunismus, Labov, Sprechakttheorie, Werkkreis, Uerden, Textsorte, NPS, Legitimationsprobleme, Aktanten, Kurztitelaufnahme, Parlamentsfragen, Textsorten, Soziolinguistik, Rawls, Uird, Textlinguistik, IPW, Positivismusstreit, Jusos, UTB, Komplexprogramms, Praxisbezug, performativen, Todorov, Namibias, Uenn, ZSta, Energiekrise, Lernzielen
- 1980s: Gorbatschows, Myanmar, Solidarnosc, FMLN, Schattenwirtschaft, Gorbatschow, Contadora, Sandinisten, Historikerstreit, Reagans, sandinistische, Postmoderne, Perestrojka, BTX, Glasnost, Zeitzeugen, Reagan, Miskito, nicaraguanischen, Madeyski, Frauenforschung, FSLN, sandinistischen, Contras, Lyotard, Fachi, Gentechnologie, UNIX, Tschernobyl, Beijing
- 1990s: BSTU, Informationsamt, Sapmo, SOEP, Tschetschenien, EGV, BMBF, OSZE, Zaig, Posllach, Oibe, Benchmarking, postkommunistischen, Reengineering, Gauck, Osterweiterung, Belarus, Tatarstan, Beitrittsgebiet, Cyberspace, Goldhagens, Treuhandanstalt, Outsourcing, Modrows, Diensteinheiten, EZB, Einigungsvertrages, Einigungsvertrag, Wessis, Einheitsaufnahme
- 2000s: MySQL, Servlet, Firefox, LFRS, Dreamweaver, iPod, Blog, Weblogs, VoIP, Weblog, Messmodells, Messmodelle, Blogs, Mozilla, Stylesheet, Nameserver, Google, Markenmanagement, JDBC, IPSEC, Bluetooth, Offshoring, ASPX, WLAN, Wikipedia, Messmodell, Praxistipp, RFID, Grin, Staroffice
One of the most intriguing tools for the Street Fighting Data Science approach is the new Google Trends interface (formerly known as Google Insights for Search). This web application allows to analyze the volume of search requests for particular keywords over time (from 2004 on). This can be very useful for evaluating product life-cycle – assuming a product or brand that is not being searched on Google is no longer relevant. Here's the result for the most important products in the Samsung Galaxy range:
For the S3 and S4 model the patterns are almost the same:
- Stage 1: a slow build-up starting in the moment on the product was first mentioned in the Internet
- Stage 2: a sudden burst at the product launch
- Stage 3: a plateau phase with additional spikes when product modifications are launched
- Stage 4: a slow decay of attention when other products have appeared
The S2 on the other hand does not have this sudden burst at launch while the Galaxy Note does not decay yet but displays multiple bursts of attention.
But in South Korea, the cycles seem quite different:
If you take a look at the relative numbers, the Galaxy Note is much stronger in South Korea and at the moment is at no. 1 of the products examined.
An interesting question is: do these patterns also hold for other mobile / smartphone brands? Here's a look at the iPhone generations as searched for by Google users:
The huge spike at the launch of the iPhone 5 hints at the most successful launch in terms of Google attention. But this doesn't say anything about the sentiment of the attention. Interestingly enough, the iPhone 5 had a first burst at the same moment the iPhone 4S has been launched. The reason for this anomany: people were expecting that Apple would be launching the iPhone 5 in Sep/Oct 2011 but then were disappointed that the Cupertino launch event was only about a iPhone 4S.
Analyses like this are especially useful at the beginning of a trend research workflow. The next steps would involve digging deeper in the patterns, taking a look at the audiences behind the data, collecting posts, tweets and news articles for the peaks in the timelines, looking for correlations of the timelines in other data sets e.g. with Google Correlate, brand tracking data or consumer surveys.
In this blogpost I presented a visualization made with R that shows how almost the whole world expresses its attention to political crises abroad. Here's another visualization with Tweets in October 2013 that referred to the Lampedusa tragedy in the Mediterranean.
But this transnational public space isn't quite as static as it seems on these images. To show how these geographical hashtag links develop over time, I analyzed the timestamps of the (geo-coded) Tweets mentioning the hashtag #lampedusa. This is the resulting animation showing the choreography of global solidarity:
The code is quite straightforward. After collecting the Tweets via the Twitter Streaming API e.g. with Pablo Barberá's R package streamR, I quantized the dates to hourly values and then calculated the animation frame by frame inspired by Jeff Hemsley's approach.
One trick that is very helpful when plotting geospatial connections with great circles is the following snippet that correctly assembles lines that cross the dateline:
for (i in 1:length(l$long)) {
inter <- gcIntermediate(c(l$long[i], l$lat[i]), c(12.6, 35.5), n=500, addStartEnd=TRUE, breakAtDateLine=TRUE)
if (length(inter) > 2) {
lines(inter, col=col, lwd=l$n[i])
} else {
lines(inter[[1]], col=col, lwd=l$n[i])
lines(inter[[2]], col=col, lwd=l$n[i])
- Beautiful Data - Musings on Big Data, Network Science and Information Visualization by Benedikt Koehler and Joerg Blumtritt


text:
" Science is organized common sense where many a beautiful theory was killed by an ugly fact". - Thomas Huxley
David Stubbs is a freelance British music journalist and author. Between 2004 and 2006 he was reviews editor for The Wire, the UK based magazine dedicated to avant garde and experimental music of all genres. Between 1987 and 1988 he was staff writer at Melody Maker, before going on to join the staff of the NME. As well as music, he also covers sport, film, literature and TV - his work regularly appears in The Guardian, Arena, The Wire, Uncut and When Saturday Comes. David's latest book is Fear of Music: Why People Get Rothko But Don't Get Stockhausen.
First broadcast on 15th January 2010.
See also here for Is Music Journalism in a Critical Condition? live event.


text:
Someone at Mountain Dew thought it would be a good idea to let Tyler, The Creator direct some online commercials for them, and amazingly, that decision somehow blew up in their faces. Tyler's series of ads revolves around a nefarious goat named Felicia who talks in Tyler's voice and who wreaks annoying havoc on anyone around. In the ad in question, Felicia is part of a police lineup, one otherwise populated exclusively by black men, as a beat-up elderly white woman freaks out with fear about identifying her attacker. This is, of course, played for laughs. As Billboard reports, a Pepsi spokesperson apologized for the ad and "said [the company] understood how the ad could be offensive." What's still not clear is who the ad could've ever appealed to. The ad has since been pulled from YouTube, but thanks to the public servants at Worldstar, you can watch it below.
Two other Tyler ads, including one where the goat attacks a waitress, remain online.
You Might Also Like
Tyler, The Creator & Erykah Badu To Guest In Black…
Watch Tyler, The Creator Do "Look At Wrist" With Father In…
Tyler, The Creator – "Diaper"
Watch Tyler, The Creator's Wolf Documentary
I don't even see how Pepsi put any of these on the air in the first place.
I feel like if they put a disclaimer at the end "Directed by Tyler, The Creator" it wouldn't have been labeled "The Most Racist Commerical of all time.
Not to mention the first ad glamorized abusing women.
You can throw up your hands and say, "well, what do you expect from Tyler, The Creator!" But, somebody at Pepsi approved the ad. I just don't know how some people have jobs when so many talented people are out of work. SMH.
I dig some of the music (Earl mostly) but holy shit these kids aren't funny at all.
adddo | Posted on May 1st, 2013 +13
"It's not racist because I'm friends with Frank Ocean, and he's black." -Tyler
It's not racist, it's just stupid. Like the Loiter Squad show. Love a lot of the OF projects musically and their antics can be amusing but these skits are just unfunny and dumb.
I always wondered how people didn't react this way to all the carl's jr. commericals…
'played for laughs'? Would help if it was funny.
0% funny at all…shame on pepsi and shame on people who think tyler the creator is an artist
I find this shit pretty funny. It's just tyler fooling around.
Hidden due to low comment rating. Click here to see
Protest their products this is unacceptable! No excuses to stereotype young African American men in this type of way. to place this out there and then pull it, the real intent was to create this ad to down the African American young men
And shame on everyone involved even those young African American men who allow them to use them to be viewed at this type of level. For he record the African American men I know are serving their country, not chasing white women and are college bound. Make a video with facts, white males who are serial killers. Know there's some facts to that. Unacceptable
Are you even black? It's just supposed to be a stupid commercial for cheap laughs. It's not a friggin political statement. It's only weird if you make it weird. Not everything has a deeper meaning.
Hidden due to low comment rating. Click here to see
ok first of all shut up u dum bitch if this had all white ppl or was about some serial killer line up with all white ppl everyone would laugh at it and say its a joke but we have ppl like u u dum hoe bag tht say tht shit so go fuck yourself its NOT raciest its just a joke you slut
I think Pepsi got exactly what it wanted…
What THE FUCK ?!?!? Pepsi just lost my business. Why was this even released in the first place? The idiot who made that descision needs to be fired. Tyler's an idiot and should not be allowed to breed. Sheesh.
These commercials are not country cool and are too extreme to do the Dew are most certainly not the choice of a new generation.
it isnt racist
cody88…get an education,you dummy! Show respect to your fellow man(person). Life is hard enough without your bullshit!Everyone needs to calm down,life is short.
Leave a ReplySign in Sign in with Facebook
You must be logged in to post, reply to, or rate a comment.


text:
This post is about the robby designed Parking distance add on The wisec kit is another product and really should be in it's own post. Maybe send a PM to kamikadze SNO
It is possible to buy system through [email protected].
The warehouse of production of WiSEC is in Moscow, Russia.
Products go any transport company on your discretion.


text:
Louis Florian Zarecki, 69, 2909 Highway 34 North, Junction City, died Sunday, Jan. 2, 2000, at his home in the town of Eau Pleine.
Services will be at 10:30 a.m. Wednesday at St. Michael Catholic Church in Junction City with the Rev. John A. Wisneski officiating. Burial will be in St. Michael Cemetery in Junction City.
Visitation will be from 4 to 8 p.m. Tuesday at the Martens/Kundinger Funeral Home and from 9:30 a.m. Wednesday at the church until the services. The parish rosary will be held at 8 p.m. Tuesday.
A memorial will be established in his name at a later date.
Mr. Zarecki was born June 20, 1930, in the town of Carson, a son of the late John and Angeline (Tylka) Zarecki.
He was married to Joann J. Polcin on June 21, 1958, at St. Michael Catholic Church in Junction City. She died Nov. 13, 1995.
As a young man, he played in a polka band and then began a 33-year career with the Portage County Highway Department while also maintaining a dairy farm. He retired from the highway department in 1991.
Survivors include four sons, Wayne (Karen) Polcin, New Philadelphia, Ohio, Chuck (Juanita) Zarecki, Junction City, and Steve Zarecki and Gary Zarecki, both of Stevens Point; three daughters, Debbie (Jeff) Gollon, Plover, Sharon (Dave) Hannas, Hooper, Wash., and Lisa (Ron) Zarecki, Junction City; three brothers, John, Stevens Point, Raymond (Florence) and Peter (Adriann), both of Junction City; two sisters, Angeline (Edward) Mallek, Junction City, and Mary (William) Houtary, Stevens Point; and 16 grandchildren.
He was also preceded in death by a daughter, Karen; one brother and three sisters.


text:
Katy Perry Makes a Convincing Argument
Katy Perry performs in Times Square. (June 15, 2009)
I love her.
this chick is so fine it makes my loins writhe in pain
I would so love to suck those nipples. Might even pay.
I adore this chick. I wonder if she needs a stalker.
Deep thought! Thanks for contibrutnig.


text:
Quality CareFind out why Mayo Clinic is the right place for your health care. Make an appointment.
Meet the StaffFind a directory of doctors and departments at all Mayo Clinic campuses. Visit now.
Research and Clinical TrialsSee how Mayo Clinic research and clinical trials advance the science of medicine and improve patient care. Explore now.
Visit Our SchoolsEducators at Mayo Clinic train tomorrow's leaders to deliver compassionate, high-value, safe patient care. Choose a degree.
Professional ServicesExplore Mayo Clinic's many resources and see jobs available for medical professionals. Get updates.
Give to Mayo ClinicHelp set a new world standard in care for people everywhere. Give now.
Mayo Clinic offers appointments in Arizona, Florida and Minnesota and at Mayo Clinic Health System locations.
Subscribe to Housecall
Our general interest e-newsletter keeps you up to date on a wide variety of health topics.
Coenzyme Q10 is a nutrient found in the body, and in some meats and seafood. Taking coenzyme Q10 supplements may reduce shortness of breath and the collection of excess fluid. However, the supplement hasn't been well studied in cardiomyopathy. Discuss the use of coenzyme Q10 or any supplement you're considering taking with your doctor to ensure that it won't interact with any medications you're currently taking.
Mayo Clinic is a not-for-profit organization. Make a difference today.
Mayo Clinic is a not-for-profit organization. Proceeds from website advertising help support our mission. Mayo Clinic does not endorse non-Mayo products and services.
Check out these best-sellers and special offers on books and newsletters from Mayo Clinic.
Any use of this site constitutes your agreement to the Terms and Conditions and Privacy Policy linked below.
A single copy of these materials may be reprinted for noncommercial personal use only. "Mayo," "Mayo Clinic," "MayoClinic.com," "EmbodyHealth," "Enhance your life," and the triple-shield Mayo Clinic logo are trademarks of Mayo Foundation for Medical Education and Research.
We comply with the HONcode standard for trustworthy health information: verify here.


text: Popcorn and buttered toast for Thanksgiving? Why not? If you're a Charlie Brown fan (and who isn't?), you'll see the wisdom of serving such fare (and actually, we know a few recovering Martha wanna-bes who see the appeal as well). Settle down with the kids, and a big bowl of the popped stuff tonight (you can skip the toast), and enjoy the Peanuts gang's holiday celebration together as you gear up for your own next week. This video is a nice way to remind everyone (including yourself) that the point of Thanksgiving -- or any celebration -- isn't necessarily the elegance of the menu or table settings, but rather the joy of gathering together and truly appreciating those around you. Now, if only we could all find a beagle who could cook, we'd be set.


text:
Royal Gardens
If you have ever stopped to admire the gardens at Royal Resorts, you'll have noticed the variety of flowering shrubs that seem to vie for attention wherever you look. Many species are native to the Yucatan Peninsula and others were introduced to the area from different parts of Mexico or the wider world. Regardless of their origin, they all share one thing, the constant and loving care of the Royal Resorts ...Read more ›


text:
|Home > Server Virtualization > News > Acronis chosen as BackupsSolution for global production agency hub+'s virtual environment
Virtualization World 365: Server virtualization, hardware consolidation, virtual machines, P2V transformation, physical to virtual transformation
Set up in 1971, hub+ has 85 employees based at its HQ in London and across its network of offices in Singapore, Melbourne and Atlanta. As a production agency it works directly with clients including O2, Barclays and Sainsbury's in the UK, to create press adverts, TV ads, produce and manage short films and interactive websites. It also works with ING and AT&T in Atlanta and Mercedes, Aviva and Singtel in Singapore. With so much data being generated the company recently virtualised its entire infrastructure to a VMware® environment to streamline its processes, reduce its hardware footprint and help manage its costs.
|Related White Papers
|Related News
|Read more News »
|Related VW365 TV
|Related Web Exclusives
|White Paper Downloads
Keep up to date with the latest industry products, services and technologies from the world's leading IT companies.
Latest IT jobs from leading companies.


text:
Hdwallpaperpics.com offers a huge collection of high resolution Sun Rise wallpapers, pictures, and photos. Please check out our wide selection of sharp, crystal clear, media wallpapers that are delivered in high definition. We have the best gallery of the latest HD Sun Rise Wallpaper, pictures, and photos to add to your PC, Mac, Iphone, Ipad, or android devicev
Are you looking for Google wallpapers? If you are, then you should definitely visit www.hdwallpaperspics.com as soon as possible. This website has amazing Google wallpapers for you to choose from. These backgrounds will really make your laptop or desktop look very nice. You can choose any wallpaper that you like. There is a wide range of options to choose from. In the end you will definitely find the wallpaper that you were looking for. There are so many wallpapers on www.hdwallpaperspics.com. If you want a Google background, then search for them on the www.hdwallpaperspics.com website so that you can find exactly what you are looking for. There are thousands of wallpapers to choose from!