Wikipedia starts work on $2.5M internet search engine project to rival Google [pdf] - <AUTHOR> https://m.wikimediafoundation.org/wiki/File:Knowledge_engine_grant_agreement.pdf <comment><USER2> This was/is actually an extremely controversial project. The corporation (basically the Executive Director) pursued the grant and the idea without soliciting input or really disclosing it to the community of editors, and eventually one of the community-elected trustees was removed for questioning the lack of transparency. The community has a long list of software improvements that they'd like to see to the core platform.

A recent employee survey showed only 10% of WMF staff approved of the Executive Director, probably in large part due to things like this.

A critical take on the project as it has been handled: [http://permalink.gmane.org/gmane.org.wikimedia.foundation/82...](http://permalink.gmane.org/gmane.org.wikimedia.foundation/82112)

[https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2...](https://en.wikipedia.org/wiki/Wikipedia:Wikipedia_Signpost/2016-01-13/In_focus) is a pretty good (but dated) overview from Wikipedia's weekly newspaper, but there's a few others in the Signpost and a few blog posts across the web.

<comment><USER3> > and eventually one of the community-elected trustees was removed for > questioning the lack of transparency

This was a hypothesis about his removal when he was initially was removed, and has since been refuted by multiple sources.

<comment><USER4> [citation needed]

</comment></comment></comment><comment><USER5> The biggest problem is the lack of data about what people are searching. It's a catch-22 that's very hard to break in the face of Google's search dominance and ubiquity.

By Google being the best, it only becomes better, and introduces a huge barrier to entry to competitors. It used to be possible to know what people were searching for to end up in a given Wikipedia article, but the process is now only asynchronous (and limited) through Webmaster tools[1]

In my mind, the most interesting aspect of the announcement should not be how much money they have to spend, but how they plan on solving this paradox.

[1] [http://webmasters.stackexchange.com/a/60350](http://webmasters.stackexchange.com/a/60350)

<comment><USER6> I think facebook should be able to build a search engine. Don't know why they don't have one yet

<comment><USER7> Facebook wants a single platform to provide the Web through. If they built a search engine, it would only search Facebook.

<comment><USER8> Facebook search is broken, twitter search works much better.

<comment><USER9> It's intentionally broken.

Despite many real (though also some exaggerated) counter-examples, Facebook does have features to protect privacy.

One of those things is you generally can't get very useful results from search unless you're friends with someone, or a friend of a friend (depending on the user's privacy settings). You can't see general trends, or even search for every person with a given first and last name in an area, for example.

It used to be more open, but they've heavily restricted the breadth of data returned from searches within the past few years.

<comment><USER10> It can't even find anything _in my own posts_. If it's a permission problem, that's a pretty fuckin' serious permission problem right there.

On top of which, Graph Search is _still_ disabled inexplicably in many regions, for reasons which were never explained.

<comment><USER11> > It can't even find anything in my own posts.

Ditto. I have seen this as a crippling deficiency in the Facebook platform. Not being able to search properly in my own posts or in the groups I'm a member of really sucks. For all the engineering prowess, open sourced tools, etc., shown by Facebook, the lack of a working search makes the company seem incompetent from the top down.

Sometime ago I started storing important information (like others links, my own comments, etc.) outside of Facebook where I can find them easily. I also started a Facebook page and added Notes into it to make it easier to document, find and share things. It seems ridiculous that I'd have to do this just to have access to information, but that's been the sad state for years.

<comment><USER12> FB search is terrible because there is no incentive for an engineering team to work on it, and in fact maybe quietly discouraged from doing so. FB make money on from the news feed, so any feature that distract users from scrolling down their feed will come up losing on an A/B test where ultimate metric is ad views and clicks.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER13> Summary of the approach (p10):

"1) Public curation mechanisms for quality;

2) Transparency, telling users exactly how the information originated;

3) Open data access to metadata, giving users the exact date source of the information;

4) Protected user privacy, with their searching protected by strict privacy controls;

5) No advertising, which assures the free flow of information and a complete separation from commercial interests;

6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach."

My first thought: How will transparency impact SEO? Will spammers be able to better game the algorithm when they know its internals?

However I am excited at the prospect of a wikipedia-like public curation system for the entire web. I admit I'm flabbergasted that the whole thing ever worked, but it does.

<comment><USER14> _1) Public curation mechanisms for quality;_

The Mozilla / open directory project tried this. Curation doesn't scale and often assumes a single unifying ontology. This is particularly problematic in a cross-cultural context. Besides, 'quality' is not a unidimensional metric in a result set: consider timeliness, authority, notability, uniqueness, comprehensibility, etc.

 _2) Transparency, telling users exactly how the information originated;_

Most search engines include a URL, I can see a [crawldate] button like the [cache] or [translate] buttons on each hit adding some information, but this will be of dubious additional utility for most searches.

 _3) Open data access to metadata, giving users the exact date source of the information;_

As above.

 _4) Protected user privacy, with their searching protected by strict privacy controls;_

We have duckduckgo already, friends are welcome but it's hardly a unique offering nor a trustworthy one given Snowden's revelations regarding the scale of systematic 5 eyes traffic monitoring/recording.

 _5) No advertising, which assures the free flow of information and a complete separation from commercial interests;_

DDG or Google or Bing with plugins can supply this. Not ground breaking.

 _6) Internalization, which emphasizes community building and the sharing of information instead of a top-down approach._

This is so amorphous as to be a non-point.

So out of six points, 2 things (33%) are only useful in edge cases, 1 thing (16%) is too vague to be useful, and the other 3 things (50%) are currently implemented by others and have been tried before.

I would like to see the input of the former Blekko guys on this, [https://news.ycombinator.com/user?id=ChuckMcM](https://news.ycombinator.com/user?id=ChuckMcM) \+ [https://news.ycombinator.com/user?id=greglindahl](https://news.ycombinator.com/user?id=greglindahl)

<comment><USER13> _> Curation doesn't scale and often assumes a single unifying ontology_

Wikipedia is a pretty big exception to that assertion. Perhaps DMOZ (a clone of Yahoo circa 1996) is not the only way to do curation. Perhaps Wikipedia could apply what has worked for Wikipedia, i.e. develop a set of POV-neutral criteria for organizing collections of links and then invite everyone to participate.

It's really easy to be negative. But that's something that might at least be an interesting research project for the #1 open-curation system in the world.

<comment><USER14> You make a fair point. I'm not rubbishing Wikipedia, just questioning the supposed USP. I would also point out in response to your argument that a Wikipedia article and a set of search results are apples and oranges.

The article is written once then modified or evolved occasionally by (almost exclusively) humans, but very frequently read. It is intended to be intelligible, being structured and based in natural language. It has a very well defined scope within a flat namespace, and often clear relations to multiple formal ontologies. It is structured to be consumed in part or in whole, and may contain rich media and strong supporting contextual information (related pages).

By contrast a search result summarizes a set of potential information sources that may answer a search query in whole or in part, to various definitions of "answer". It is generally written once, by a computer, and thrown away after some period of caching. It is intended to be concise. Each component result has relatively poor context, relying upon the searcher to interpret timeliness, authority, notability, uniqueness, comprehensibility, etc. with the limited information presented, typically a very short content excerpt. It is structured to be scanned, classically in a ranked fashion from "best hit" to "worst hit", and is generally a wall of text.

Wikipedia successfully attracts people to contribute to the former, but the latter - where the information product is generated on the fly and lasting impact is amorphous (nothing particularly concrete for contributors to point to and say "I did that! Warm and fuzzies!") - is a very different beast.

I too believe there is room for innovation ... there are potentially low hanging fruit like inter-linguistic semantic queries (not keyword search) ... but there are no such key problem areas identified in the paper's summary.

<comment><USER15> The other big problem is that curating search results is inherently about prioritising a _position_ rather than establishing a sourced and reasonably neutral version of the truth.

I'm imagining the edit wars and debates that take place on contentious wordings or facts in some parts of Wikipedia, but on a much wider scale involving hundreds of SEO consultants each aware that changing a particular criterion will have a quantifiable impact on their clients' bottom line. It doesn't sound like it would be fun to police.

<comment><USER13> Wikipedia already curates links to some extent on every page under "External Links". So there is a seed there.

And even the page text is not immune from the problem you describe. Grading and prioritizing sources is a fundamental part of producing a "reasonably neutral version of the truth." It's what determines what gets cited and how prominently it influences the article.

So while I wouldn't equate text and links in terms of the difficulty of managing POV-neutrality, I would say they sit on a spectrum.

</comment></comment></comment></comment></comment></comment><comment><USER16> This is a very, very, very small amount of money if you want to build a search engine, let alone one "to rival Google" (source?). Looks like the goals are realistic, though - look how wikipedia search could be extended beyond results from wikipedia.org, build some test sets. And get a better idea what it really is that is supposed to be built.

<comment><USER17> I run a knowledge engine project that actively mines facts from web-based sources and third-party data dumps. It was featured on the front page of HN a while ago, and has a total of $10 in funding (from a single donation; not a typo). I have, however, put a ton of time into it, and it's something I'm very passionate about. I'm fairly confident Wikipedia can have success making initial headway on their grant objectives with $250,000.

If anyone's interested, here's a demo of my own project: [https://tuvalie.com/fae/?q=Albert%20Einstein](https://tuvalie.com/fae/?q=Albert%20Einstein)

<comment><USER18> Over 15 years ago, for my undergraduate thesis I set up a "Hypermedia Textbook" on the history of my field. I had to manually collate all the info, manually scan in every photo, and type in every last bit of text and html. The end result was a couple hundred pages that looked very, very similar to what your Einstein page looks like! At the time, I knew a better way would emerge, but didn't know how or when. It's moments like this that I (a) feel old :( and (b) am amazed by the times we live in and the speed at which things are happening! :) Thank you for providing such a wonderful, if unintentional, moment of self-reflection!

<comment><USER17> Thank you for checking it out! And if you have any ideas for how things can be improved, I'd love to hear them :)

</comment></comment></comment></comment><comment><USER19> Weird; they have anual drives to raise money to keep the site running; would not expect that the'd have 2.5m lying around to do pet projects like this...

<comment><USER20> Most non-profits, especially the ones that are always asking, usually have a lot of funds.

Before I give, I go to guidestar, hit free preview(they try to trick you into a paying membership), download last few years of 1040's, and see if everything looks copacetic. I look at who is making the most money. Their is usually one person making a very good living. California non-profits are much easier to scrutinize than Deleware non-profits.

<comment><USER21> They are required to give you the 990 form if you email them usually.

In anycase, [https://wikimediafoundation.org/wiki/Financial_reports](https://wikimediafoundation.org/wiki/Financial_reports)

And the internet archive is a lot more deserving.

<comment><USER22> I believe in internet's archive mission even if I don't use the site that often but I use wikipedia too much, I can't justify not donating to them when they need it.

</comment></comment></comment></comment><comment> seven-dev I support it 100%. I love Google but they have too much power and I'm sure they'll start taking advantage of that soon (like they did with Google+ and Youtube).

<comment><USER23> I can't see it being much better I find Wikipedia impartiality lacking it's very US-centric.

For example something like the history of the Alaskan panhandle as seen from the US perspective is totally different when seen from a Canadian perspective.

I never use Wikipedia as a primary source of info even the linked sources I try to use as least three independent sources.

I would certainly like to see "Reliable and trustworthy information" but who do I trust who is reliable?

<comment><USER24> I don't know if you can trust anyone, I mean bias is everywhere. You can pick a side.

For example the Portuguese Wikipedia, shared by Brazil, Portugal (and other countries), with controversial matters many on colonization in the last 500 years, where both sides have academic work to support their contradictory views. Which views prevail?

An example of things being done differently are some Ex-Yugoslav countries (Serbia, Croatia, Bosnia, Montenegro, etc) whose languages are more similar than Portuguese and Brasilian Portuguese, and each one has its own Wikipedia, with different articles on the same subject depending on their point of view. Lately, I've been seeing more of the Serbocroatian Wikipedia, which I think aims to unite more of the others.

I don't know which way is better, I'm just a user.

Another reason you can't trust anyone, and this is general to the Internet, is that shilling, commercial and political interests aiming to change perception are everywhere. On reddit or facebook, with or without sources. It's the worst aspect of the internet for me these days.

</comment></comment></comment><comment><USER25> The title is misleading. It's 250k, not 2.5m and the goal is a knowledge engine, not a search engine.

<comment><USER26> Not only is it a search engine, but it is a grant application that has had WMF staff leaving in droves, and has greatly upset many, many others - who will quite likely also leave.

It's very, very sad. And it's also a shameful moment for the WMF.

 _edit:_ and don't just think it's me saying it. The WMF has had a mass exodus of staff in the last week or so. If you speak to any WMF non-executive staff members directly, you'll quickly find out that morale is at an all time low, and confidence in the WMF Board is sitting at something like 12%.

<comment><USER27> Can you say more/explain? What about the application is so upsetting? What is shameful about this?

The Knight Foundation is about as upstanding as you can get, so it can't be that (full disclosure, I've received funding from them, so I'm definitely not unbiased on that point). So, what exactly is it that's so shameful here?

<comment><USER26> To be clear: my issue (and in fact, most peoples' issues) are not with the Knight Foundation. In fact, they appear to have been above board in every way in this whole debacle. It is the WMF board who are the problem here.

See my comment here for just a few comments on this issue:

[https://news.ycombinator.com/item?id=11101262](https://news.ycombinator.com/item?id=11101262)

Frankly, there's a lot more - to understand the issue better you might want to read Liam Wyatt's blog posts:

[http://wittylama.com/2016/01/08/strategy-and- controversy/](http://wittylama.com/2016/01/08/strategy-and-controversy/)

and here:

[http://wittylama.com/2016/01/30/strategy-controversy- part-2/](http://wittylama.com/2016/01/30/strategy-controversy-part-2/)

<comment><USER27> Thanks – I appreciate the references

<comment><USER26> That's OK, ironically it was Wikipedia that taught me to always back up my statements with references :-)

<comment><USER27> As an aside, I think the internet is simultaneously great at spreading absolute bs and disinformation and pushing people to have citations handy... paradoxically, both seem to be getting more frequent. (It all depends on where you browse, clearly).

<comment><USER26> Yeah, nobody knows this more than myself. I created [citation needed] and I've watched it be misused for years. I _am_ glad I came up with the idea, but I'm resigned to the fact that it's human nature to misuse a valuable idea.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER28> This is exciting. Recently I started to work on an answer engine/search engine. It still sucks but it's a good project to work on when bored

[http://kairos.xyz/](http://kairos.xyz/)

In a few weeks I'll publish the source code and do a Show HN.

I wish a lot of luck to

[https://www.mojeek.com/](https://www.mojeek.com/)

and

[http://www.lexxe.com/](http://www.lexxe.com/)

too. DuckDuckGo also started to crawl the web with its own bot (right now they're using Yandex's api).

We need more competition from different countries. Just think about the censorship done by Baidu or how Google never plays by its own rules.

It's also interesting to think about a way to monetize a search engine. For kairos.xyz I was thinking about paid accounts (1 euro per month) providing more features, like the ability to search from the command line. For example you write "kairos Richard Stallman" and it prints basic information about Richard Stallman on your terminal.

<comment> seven-dev (Default nginx page showing up on your website)

<comment><USER28> It works for me. Care to show me a screenshot?

<comment><USER29> The host (assuming the same host) is responding with a different website when accessed via IPv4 vs IPv6.

      $ curl -4s http://kairos.xyz/ | grep title
          <title>Kairos</title>

      $ curl -6s http://kairos.xyz/ | grep title
      <title>Welcome to nginx on Debian!</title>

      $ host kairos.xyz
      kairos.xyz has address 126.96.36.199
      kairos.xyz has IPv6 address 2604:180:0:a54::24d9

<comment><USER28> Thanks, I found the problem. I thought that with Nginx, Ipv6 would just work but I had to add

        listen 80;
        listen [::]:80;

to my server block.

</comment></comment></comment></comment></comment><comment><USER30> Link to Wikimedia's wiki page on the project includes a decent FAQ: [https://meta.wikimedia.org/wiki/Knowledge_Engine](https://meta.wikimedia.org/wiki/Knowledge_Engine)

</comment><comment><USER31> Anyone want to hazard a guess at the technology they plan to implement to get this started?

Surely this is not designed to be written from scratch, so..

\- Are they using known lexical & semantic scanners? \- Is it focused on English language first? \- What crawlers will scan content? \- I'll assume it's an open platform, but license for contributors? \- What database architecture will hold the graph? \- How does it know the mark of authority, and is this primarily based on human input learning or machine learning?

I'm sure $2.5M wont touch the sides, but maybe if it's a well directed project, with healthy user contribution, based on interesting technologies they might develop a good backbone architecture. Ambitious for sure.

</comment><comment><USER32> This is a link to the actual pdf:

[https://upload.wikimedia.org/wikipedia/foundation/a/a7/Knowl...](https://upload.wikimedia.org/wikipedia/foundation/a/a7/Knowledge_engine_grant_agreement.pdf)

</comment><comment><USER33> Computing is so cheap now, google isn't going to be so dominant on text search for long. Their money is needed for video and pictures and audio, but the text internet can be cached whole by small entities now.

Maybe Wikipedia should launch a video encyclopedia to try to provide a 5 minute video of every article, for people who like videos more than reading.

</comment><comment><USER34> Doesn't it say 250k in the letter?

</comment><comment><USER35> The grant amount is $250,000

<comment><USER36> Correct. As per page 9 the most of the budget allocated for the project comes from wikimedia themselves, totaling 2,445,873.00 USD in for the fiscal year 2015-2016.

<comment> mh- do these funds come from the donations they solicit on wikipedia.org?

<comment><USER37> Yes, it's the main (only?) source of income for the Wikimedia Foundation.

<comment><USER38> There are substantial donations from other foundations and company match programs (Huuuge page):

[https://annual.wikimedia.org/2014/#s-5](https://annual.wikimedia.org/2014/#s-5)

I guess the foundations aren't reasonably a response to the Jimmy banners, the company matches probably are.

</comment></comment></comment></comment></comment><comment><USER39> Misleading title. I don't think they want the result of the grant to rival Google

</comment><comment><USER40> I am financing a $10 project to rival Tesla.

</comment><comment><USER41> I'm not kidding when I say that if they want to know where to spend the $2.5m I would start with cleaning up their core codebase. IMO Mediawiki open source code is a disaster.

EDIT: Not because it's written in PHP. Because it's architected poorly.

<comment><USER26> It's funny you should mention that. That was a point that apparently a number of WMF staff expressed, and it was apparently ignored.

<comment><USER42> There used to be a team dedicated to making MW Core better / cleaner, but that was lost in a re-org earlier in 2015.

<comment><USER26> That's a darned shame. I'm aware that there are a _lot_ of areas that people want to fix on MW Core.

I get really concerned when I hear that the person who holds the vision and direction for the Wikimedia Foundation didn't really participate in it beforehand, and I get even more concerned when I see that she branches off into proposals for search technology that appear to be far outside the scope of Wikimedia projects.

Nobody has ever thought search in Wikipedia or the various projects was particularly effective. However, bringing everything together doesn't just involve searching, and frankly there are a number of more pressing governance and community issues that need to be managed.

Perhaps I'm being a bit unfair here, but she was profiled when she first joined the WMF Board, and the following was said about her:

 _At the meeting, she described the impact on friends and family of the Chernobyl nuclear disaster, and the difficulty of getting reliable information in the face of "so much secrecy."_

Yet we see that this is _precisely_ what happened with this grant proposal. A major grant was applied for and awarded and _not even WMF staffers_ knew about it. You can see on the mailing list that it was a total shock when it was finally revealed.

I'm watching this train wreck from afar, but closer than others because some of my friends are deeply involved in Wikipedia and the WMF. I'm always amazed that a leadership change can complete kill an organisation. I've seen it in the corporate world, and I see it all the time in the volunteer world as well. The Wikimedia Foundation seems to be yet another victim of the appointment of a clueless leader, with no experience in the area or with the group they are meant to be leading, thrashing around, making changes without really understanding how systems work, the history of the organisation or relying on the experience and sage advise of the many expert and dedicated people around them, ultimately leading to a great deal of unnecessary turmoil, ill-will and frankly destruction in their wake.

</comment></comment></comment></comment><comment><USER43> If nothing else, I hope it improves the currently abysmal search features for Wikipedia today.

<comment><USER44> turns out that is the only thing this project is about. There is no web crawler, there is no external content. The grant and the money WMF is spending are being spent to improve internal search at wikipedia.

</comment></comment><comment><USER45> Good luck. We definitely need more search engines. (That Google announced to lower the PageRank(R)/site score for non-HTTPS sites is a clear indicator that they about to cross the line (monopoly). And no, DDG and most others are "just" _meta search engines_ that rely on Yahoo Boss ($$$) which future is uncertain and relies on Bing.)

There was "Wikia Search" by Wikipedia founder Jimmy Wales:

" _Wikia Search was a short-lived free and open-source Web search engine launched by Wikia, a for-profit wiki-hosting company founded in late 2004 by Jimmy Wales and Angela Beesley.

Wikia Search followed other experiments by Wikia into search engine technology and officially launched as a "public alpha" on January 7, 2008. The roll-out version of the search interface was widely criticized by reviewers in mainstream media. After failing to attract an audience, the site closed by 2009._"

[https://en.wikipedia.org/wiki/Wikia_Search](https://en.wikipedia.org/wiki/Wikia_Search)

I used Wikia Search back then, it was good enough (like Bing in comparison to Google back then).

It was based on Apache Nutch and Solr/(Hadoop(?)/Lucene ...

Maybe you can rely on Lucene or SphinxSearch projects to kick-start.

<comment><USER46> Google doesn't want to lower the PageRank of http sites, it just wants to use http vs https as a feature in ranking. That isn't particularly surprising, I would be willing to bet google already uses hundreds of such features (one of which might be PageRank).

</comment></comment><comment><USER47> Given the terrible state of advertising, I would welcome a search engine that penalizes pages with popovers, animated ads, auto playing audio, and so on. Google would never build this, given its business model.

I hope Wikipedia brings some innovation to search, untethered from advertising revenue.

<comment><USER48> Google does do this AFAIK. They dont get any money when people turn on adblock and have always leaned towards simple ads as well as filtering out sites with crummy ux in search

</comment></comment><comment><USER49> As a former NLP engineer and former wikiHow engineer, I have some perspective on this. Google has included more and more information from Wikipedia. Furthermore, Google includes snippets of external websites in the knowledge box on more and more pages.

How long will be until can algorithmically generate its own Wikipedia articles? Wikipedia relies upon coming to its site for contributions and donations. Without search, Wikipedia risks being subsumed by Google. They have a difficult position of thinking about the future without pissing off Google.

Computers are getting more and more powerful. Wikipedia needs to do stay relevant. I think this is the right decision.

</comment><comment><USER50> I don't know how should I feel about all those donation campaigns they usually do after this.

<comment><USER51> Looks like this is a grant specifically for this work (at least in part.)

I think this is an incredibly good use of their money. Google is the world's biggest surveillance machine, I hope that Wikipedia can do to them what they've already done to Encarta.

</comment></comment><comment><USER52> I've read a log of the inside wikimedia links here, and I'm confused about all the talk of gnashing of teeth and rending of cloth. This is controversial because some want to pay down technical debt rather than have a small team do knowledge graph search?

Okay...

<comment><USER26> It's not a small team compared to the size if the organisation. And you mischaracterise the situation: this is a problem with engagement, transparency and openness.

<comment><USER52> Yeah the relative merits of the initiative seem to be besides the point. If you have a toxic environment, even a proposal to cure all disease for everyone for free will attract derision.

It's hard to get worked up over some other team's morale. If its such a crappy place, just quit. The could probably literally go across the street and get a new job. I don't really care. It's all way too inside baseball for me.

<comment><USER26> You cared enough to comment. This wasn't just a place of work for me, I volunteered my time because I believed in what they were doing.

 _You_ might never have contributed in a significant fashion to Wikipedia and other WMF projects, but I did. Sure, I didn't get employed, but then again I know a lot of people I met and have continued to be friends with who are still deeply involved. _You_ may not care about your friends' morale, and you might think it's easy for people to "go across the street and get a new job", but then you seem like a pretty thoughtless person.

Of course, you've not understood at all what the larger issues are. You must have a bit of a comprehension issue, because I supplied quite a few links that you apparently read that explained the underlying problems.

Just remember though: you don't care :-)

<comment><USER52> 0) Stop with the personal attacks. It's rather unbecoming, and I don't appreciate it.

1) Being genuinely confused about what the big deal isn't the same thing as caring. It's asking for confirmation of a conclusion.

2) If you're in a situation where you're unhappy, then you have a responsibility to make yourself happy. Staying around in a crappy situation and whining about doesn't help, and neither does insulting people.

3) Wikimedia is in San Francisco. If I had to take a guess, I would say there was a literally a hundred other tech organizations in that city alone, including nonprofit organizations with a societal purpose. 18F comes to mind. Again, See 2.

<comment><USER26> 0) you don't seem to realise how you come across

1) you literally wrote "I don't really care".

2) they aren't whining. Saying so is pretty much a personal attack. It's certainly insulting. I don't appreciate it, and I'd say neither do they. Funny how that works both ways.

But interestingly enough, as has been pointed out already - people ARE leaving in droves.

I'm no longer involved in Wikipedia, but I can still be unhappy with the direction they are taking.

3) if you think that just leaving a non-profit you have emotionally invested in is an easy decision, then you really haven't thought things through. If you think it's elementary to just step out of one job and into another, that's also thoughtless.

</comment></comment></comment></comment></comment></comment><comment><USER53> That's how much 5 qualified software engineers would cost to employ for a year (gross, including compensation, benefits, payroll taxes, office space, hardware, etc, and that's on the low end of the range). Good luck with that.

<comment><USER54> It costs half a million to employ one software engineer for a year?

<comment><USER53> A software engineer qualified to work on this kind of thing is worth about $350K in combined compensation on the market right now. Typically half of that is base, while the other half is stock and other taxable benefits. The number can be higher. This is the cost of just compensation to the company, excluding the payroll tax. You can, of course, find someone a lot cheaper, but then you'd be a fool to expect the result to be anywhere near as good as what Google can pull off, because if that someone could do what Google can, why would she work for half the compensation instead of applying to Google or FB or whoever pays competitive salaries these days.

</comment></comment></comment><comment><USER55> Remember this?

[https://evolvingtrends.wordpress.com/2006/06/26/wikipedia-30...](https://evolvingtrends.wordpress.com/2006/06/26/wikipedia-30-the- end-of-google/)

</comment><comment><USER56> As a former Wikia employee, I am somewhat of a MediaWiki insider. I sped Wikia's search engine up by several orders of magnitude and then went on to pilot a number of NLP/machine learning initiatives in the company.

Jimmy Wales' already tried to make a "Google Killer" ten years ago. It was tilting at windmills to say the least. Letting individuals help manage algorithmic search results was harder than you could imagine. Let's not even get into the difficulty of building an effective crawler.

One of Wikia's former CEOs, Gil Penchina, notoriously undervalued search as a result of this very public gaffe. By the time I came in, it took over five seconds to do a simple on-wiki search. Searching across wikis took so long they actually just sent the search to Google and had you abandon the site. I personally fixed a lot of these problems, and that part was pretty cool.

So now let's get to the subject at hand, which is a search feature based on an authoritative knowledge graph. Something like this should adequately surface factual information in an intuitive manner -- optimally based on natural language. Wikia already tried this, too. They brought on a very seasoned advisor who played a crucial role in the semantic web movement far back into the early oughts. I remember going to semantic web meetups in Austin when I was in grad school quite some time ago now to hear this guy talk.

This guy was essentially the SF-based manager or lead for a small team located in Poland whose job it was to take some of the "structured data" at Wikia and attempt to build some kind of knowledge graph on top of it. This project was unsuccessful.

So why did it fail? We'll start with a lack of product direction. Wikia had and probably still has a very junior product organization that is mostly interested in the site's UI and (recently) a focus on "fandom" (yuck). The team allocated to the project was based in Poland (Poznan, to be exact), and primarily kids coming out of a technical school on their first job. Your assumption about communication being a problem would be correct. However, the subject matter expert was so entrenched in his area of specialization, the problem was even more compounded on the native English-speaker side. There was too much getting in the weeds, and not enough focus on incremental progress.

To make things worse, they tried using a proprietary, not-ready-for-primetime data store because it most closely matched the SME's preconceptions on how the data should be structured. There was absolutely not an existing business use case for this data store, and problems getting it to work turned even building a simple demo into a death march.

Either way, what I'm saying is, $250,000 is not enough to solve this problem. We have attempted to solve this problem before in the MediaWiki world. It's not going to magically get better. To make something like this work, you need:

1) Best-in-class UX people who would know how a knowledge graph provides a significant improvement over existing solutions 2) Leadership that can bridge the gap between SMEs and implementers 3) Very skilled engineering resources with backgrounds in less conventional technologies

This is a massive investment that no one is willing to spend on what is essentially a media play.

About six months later, I had built a proof-of-concept that sucked data out of MediaWiki Infobox templates into Neo4j, a well supported graph database. I was able to answer questions like, "Which cartoon characters are rabbits", and "What movie won the most Oscars in 1968" using the Cypher query language.

At that point in time, Wikia had decided they were tired of investing in structured data, and wanted to re-skin the site for a third time in as many years to make it look more like BuzzFeed.

Structured data is cool. In many cases, unsupervised learning may be what you're actually looking for. But in the end it has to satisfy a real user's needs.

Wikipedia has five million English articles. Wikia has over 20 million. As far as capitalizing on this wealth of knowledge, the devil is truly in the details. But it's a real shame that all of that information isn't put to better use than to encourage the socially maladjusted to take quizzes over which anime character they're more like.

<comment><USER57> How did you arrive at 20 million? This sounds like one of those "technically true" facts that are cooked up for investors. [http://wikis.wikia.com/wiki/List_of_Wikia_wikis](http://wikis.wikia.com/wiki/List_of_Wikia_wikis) puts the combined total of the top 1,000 wikis (in all languages) at 12.4m.

<comment><USER56> 20 million pages, not wikis -- sorry if I mistyped?

<comment><USER57> There aren't 20 million pages. Read my comment again.

<comment><USER56> There are over 300,000 wikis. I usually worked with the top ten thousand English wikis, which had over 15 million pages.

Or I'm just making the number up. Doesn't really matter to me.

</comment></comment></comment></comment></comment><comment><USER58> Google's advantage isn't just that they were first, or that their algorithm is the best- it's the CPU resources they have available to keep their data updated faster.

Search for any news item and you'll have all articles published more than 2 minutes ago included in your results, all blog posts, everything. They consume it all, and offer the output in near-real-time.

Wikimedia don't have the resources to do that. And they especially won't without advertising to pay for it.

</comment><comment><USER59> Wikipedia main asset seems based around user contributions and human interactions not programming or hard algorithms, this seems quite a leap into another field with not much money.

Bing cost MS $5.5 Billion in their field of expertise

[http://www.geek.com/news/bing-has-cost- microsoft-5-5-billion...](http://www.geek.com/news/bing-has-cost- microsoft-5-5-billion-since-launch-1423117/)

</comment><comment><USER60> This is like competing with Intel in CPU server market, where he has 98% market share.

So they are trying to compete using 2.5 mil dollars with software backed by multi billion dollars, hundreds thousands of servers, tons of data, thousands of developers, ML integration etc.?

Good luck with that. Many tried backed by x times more resources than this 2.5 mil, unfortunately all failed.

</comment><comment><USER61> I am guessing this has a different focus than their previous attempt at making a search engine, wikia search, which they abandoned fairly quickly [https://en.wikipedia.org/wiki/Wikia_Search](https://en.wikipedia.org/wiki/Wikia_Search)

<comment><USER62> Wikia is not part of the Wikimedia.

[https://en.wikipedia.org/wiki/Wikia#Relationship_with_Wikipe...](https://en.wikipedia.org/wiki/Wikia#Relationship_with_Wikipedia)

[http://community.wikia.com/wiki/Help:Wikimedia](http://community.wikia.com/wiki/Help:Wikimedia)

(Edit: Who decided that enter is not equal to enter?)

</comment></comment><comment><USER63> This is great. It feels like we live in an information overload era opposite of North Korea.

Search "Are cookies really bad for me" and find an answer that supports what you want to hear.

"Live a little" Sponsored by Nesthouse Cookies INC

</comment><comment><USER64> Comparison point: it's Bing's budget for about 5 hours.

</comment><comment><USER65> The link headline is highly editorialized, there is no mention of "google" or "rivalry" in the pdf in the link.

</comment><comment><USER66> So this is why they've been asking for donations? Made it seem like they're on the ropes.

</comment><comment><USER67> most of my google searches include a wikipedia result on the first page. I would estimate this could reduce Google's web search revenue by upwards of 40% worldwide.

<comment><USER68> Google doesn't make that much money on research quieries. It's mostly your other queries are what enables them to sell ads.

<comment><USER67> well nearly everything one might conceivably google has a wikipedia page...

</comment></comment></comment><comment><USER69> a JV might be a better idea...$2.5M isnt that much money and i doubt it will even come close to being useful relative to the other search engines

</comment><comment><USER70> Surprised no one has been mentioning qwant.com yet.

</comment><comment><USER71> $250000 != $2.5M

<comment><USER26> Part of a 2.5 million dollar grant, this is the first stage.

</comment></comment><comment><USER72> I support it 100%, but still think they should use search advertising to cover costs and further development instead of asking for donations every year.

especially if they can make something that actually does rival google... other companies have spent billions and not gotten very close.

</comment><comment><USER73> Google already _is_ the search engine for Wikipedia. And Wikipedia is the content provider for Google. Why mess up such a beautiful arrangement? [http://newslines.org/blog/google-and-wikipedia-best- friends-...](http://newslines.org/blog/google-and-wikipedia-best-friends- forever/)</comment>


Deep Learning Machine Solves the Cocktail Party Problem - <AUTHOR> http://www.technologyreview.com/view/537101/deep-learning-machine-solves-the-cocktail-party-problem/ <comment><USER2> Audio samples of the results would be nice, but I didn't find any links in the pdf

My guess would be that extracting useful audio is possible, but the idea evoked by the paper's title "Deep Karaoke", of flawlessly removing vocals from a finished song, is a lot harder, or maybe impossible. Anyway this has many useful applications</comment>


The Lost Quarter-Century in Data Compression - <AUTHOR> http://blog.podly.tv/the-lost-quarter-century-in-data-compression

<comment><USER2> Huffman coding is computationally much simpler, take a histogram of the input tokens and make a token->bitstream function. At the time it was released a newish computer would be an IBM 650 with 2000 words of memory on the drum and 60 words in core. (Note that absence of 'k', 'm', or 'g' in those numbers.) It was "quite popular". 1800 of them were sold.

The LZ algorithms of the later '70s are more computationally involved with dictionary lookups in mutating dictionaries. At this time a newish computer would be a PDP-11 with 64k words of memory and a hard disc drive that could store megabytes! I'm not sure how many were sold, but universities were littered with them.

I suggest LZ style algorithms merely waited for appropriate physical tools.

<comment><AUTHOR> It's more the rule than the exception when research is done on ideas that at the time of publication are impractical to manifest as a working machine or program. To pick just one (possibly the most well known) example: Einstein's paper on the special theory of relativity was published in 1905. The most significant outcome of this paper took 40 years to manifest itself. Einstein certainly wasn't waiting for appropriate physical tools.

Lempel and Ziv's critical insight - that modelling and coding of probabilities are distinct and separable - could have been understood decades earlier. It wouldn't have mattered that their research couldn't be immediately turned into a working tool. Information Theory would have benefitted and we would surely be further along today.

But, yes, it is certainly true that advances in technology give us an incentive to ask questions that had previously been ignored or misunderstood.

<comment><USER3> Computer science seems actually to be the exception. Yes, though the insight could have been had earlier, it somehow wasn't. And this is common in computer science. I now attempt to support this thesis.

Even the stereotypical "impractical" research algorithms (with efficient complexity but huge constants) are _implemented_. That's the magic of computer science, that you can breath life into abstract ideas. But if you cannot implement it at all, there are millions of other cool ideas that you can implement, so you may gravitate towards them. (another argument is that if you can't implement it, it's classed as pure maths. And if you can't prove it, or define it clearly as a conjecture, I'm not sure what it is or what you could do with it - perhaps futurism/science fiction, which has its place.)

The value of implementation is due to computer science ideas tending to be complex and abstract - but once you have an actual concrete working model that you can play with, they can become almost trivial to understand. And so, comparatively, more progress in research is made based on working code than on purely theoretical ideas. It's so _much_ easier to find mistakes in an idea if you can build it. And once your theory is embodied in living, working code, it's tremendously easier to reason from that as a base; much as it is easier to see what is over the next hill after climbing it.

Hardware is closer to your idea of incentive: the inventor of the Wallace tree multiplier told me that it was a simple and obvious idea, but it wasted a lot of silicon; and no one entertained that kind of waste. It simply wasn't conceivable. Perhaps this is due to the incredible rate of improvement in silicon (Moore's Law), so that it's psychological difficult to appreciate (or even believe) that your iPhone will be 1000 times faster in 15 years; or what changes that even means for algorithms. Perhaps the most basic assumptions of the algorithm changes? When x10 quantitative improvement often translates into a qualitative improvement, what does a x1000 change mean? Too hard to predict.

<comment><AUTHOR> Computer science gives us a better opportunity to test ideas than, say, particle physics. The ability to experiment so easily is a major part of the appeal for many of us.

> Even the stereotypical "impractical" research algorithms > (with efficient > complexity but huge constants) are > implemented.

Well, they're all Turing machines, but you're not going to find any video processing algorithms that were implemented on a 4K Altair so let's not go too far.

It's interesting to consider a given novel idea and wonder how far back in time it would have been possible to implement it in a practical way and how much further back in time it would have been useful simply to know how it works.

Let me pick a concrete example that we are all familiar with: Hashing. Hashing and hash lookup algorithms have been around a long time. It's an area that people might expect would have been exhausted of significant new ideas long ago. Not so! Cuckoo hashing was invented in 2001 and gives us dramatically more efficient use of memory at a slight performance cost. It can make the difference between a hash table that fits in memory and one that doesn't so it's a meaningful advance.

Cuckoo hashing could have been implemented on a 4k Altair and it would have been useful there. But no one had the idea then.

But wait-- It gets stranger! Cuckoo hashing was literally under our noses and somehow no one saw it! It is a variation of set associative hashing which is what most CPUs had already been doing in their caches for years. It's not exactly the same algorithm but if you check you'll see it's so close the distinctions are minor.

Fascinating, isn't it? Here is an important algorithm that was actually live and running in hardware yet no one considered building it in software for a long time.

<comment><USER2> Isn't the last part of the Cuckoo hashing paper the part that documents it as slight less good compared to existing algorithms? I mean it never comes out and says it, but if you look at the pictures… I think Cuckoo's best point was the provability of its bounds, but it has been a while since I read that paper.

I like the example of Parsing Expression Grammars. The idea of memoized top down parsing drifted past back in the 60's and early 70's when the folks like Aho and Ullman were busy slaying wild streams of symbols with ever more clever algorithms but was dismissed because of its memory footprint. Then in 2002 or so Bryan Ford said " _Oi! What do you mean too much memory? My PC has an L2 cache bigger than that._ "[1] Now PEGs are widely used and I must say (having built dynamic grammar LR systems back when my time shared vax had a main memory the size of the L2 cache in my iMac) especially useful for dynamic grammars.

[1] Being an entirely fabricated quote, there is no applicable reference for this quotation.

<comment><AUTHOR> > Isn't the last part of the Cuckoo hashing paper the > part that documents it > as slight less good compared > to existing algorithms?

No, Cuckoo hashing is a trade of a little run-time performance for better memory utilization. It's still O(1). The performance cost amortizes as a constant.

If you have gobs of memory you don't need it. But if you're memory-constrained it's a big win.

</comment></comment></comment></comment></comment></comment><comment><USER4> If you're interested in another example of how success breeds stagnation, see _On Proof and Progress in Mathematics_ by William Turston, available at [http://www.ams.org/journals/bull/1994-30-02/S0273-0979-1994-...](http://www.ams.org/journals/bull/1994-30-02/S0273-0979-1994-00502-6/S0273-0979-1994-00502-6.pdf)

In particular look at section 6 where he describes how his research into foliations had the effect of killing the field for a number of years. He contrasts this to how his different approach to geometrization of 3-manifolds lead to the healthy growth of the field. By, among other things, his efforts at exposition, and his deliberate withholding of results he already knew the answer to so that other people could prove them instead.

<comment><USER5> _his deliberate withholding of results he already knew the answer to so that other people could prove them instead_

That's extraordinary! I never heard of such a thing. Are other cases known?

<comment><USER6> Gauss seems to have proved lots of (surely publishable) things that he didn't publish. I don't know what his reasons were.

<comment><USER7> According to Erdos _, Gauss was very discouraging to his students, because when they came and told him that they'd proven something, he would tell them he'd proven it years before.

_ from "The man who loved only numbers.' by Paul Hoffman.

</comment></comment></comment></comment><comment><USER8> Silly.

Practical dictionary-based schemes have an entropy encoder on the back end and often that's a Huffman encoder or something similar in spirit.

Huffman encoding is superseded not by dictionary-based methods, but by arithmetic encoding, which can compress beyond the Huffman limit because it can use fractions of a bit.

7zip, for instance, beats the pants off gzip because it uses a very large dictionary window AND arithmetic compression instead of huffman encoding on the back end.

<comment><AUTHOR> I think you're missing the point.

The article does not claim that Huffman coding was superseded by dictionary methods. It does not confuse statistical modelling with encoding.

</comment></comment><comment><USER9> I remember reading an interview with Phil Katz (pkzip) like a decade ago about how they were looking next at markov models for more advanced compression. It even made me go to the library to look up "markov".

So over a decade later, nothing really new.</comment>


Forecasting Bitcoin Price with LSTM - <AUTHOR> https://towardsdatascience.com/lstm-for-time-series-prediction-de8aeb26f2ca <comment><USER2> Interesting, kind of guessed the answer was no or the author would have retired instead of writing the article...</comment>


You Could Do Almost Anything Part 1 - <AUTHOR> http://www.elischiff.com/blog/2016/4/12/do-almost-anything <comment><USER2> > Instead, the logo is noteworthy precisely because it has achieved critical > acclaim despite, or more accurately, because of its failure to communicate.

That, right there, is a damning indictment of the entire modernist project: communication achieves acclaim because it fails to communicate; art achieves acclaim because it fails to be beautiful; life achieves acclaim because it is sterile.

As an aside, I love the dry tone of the piece, with lines like 'However, not only does the new abstract logo break this entire history, but it adds even more potential readings of the logo' — with a demonstration of how the upside- down new logo looks like a hand flipping the bird. The essay's full of bone- dry commentary like that. It's glorious in its restrained vehemence.

<comment><USER3> I think the primary objective of a logo is to be a recognizable symbol for an organization or concept. It's not entirely clear to me that to achieve that objective the logo must fully describe the entity or be some identifiable representation of what that entity does.

For example Apple's logo speaks very little to what it does, and to a greater or lesser extent so do many others including Uber (new and old), Xiaomi, Square and Palantir just to pick a few of the largest startup-style companies in the last few years. Of course the logos come to work as a brand and we identify them with those entities as we get to know the brands.

Under this criteria, isn't the objective of a logo to simply be recognizable? In that case, kind of in a self-fulfilling manner, doesn't even a critical article such as this one make it more successful?

Personally I like the logo, and I found it interesting to hear the logic behind it. Sure it's not to everyone's liking, but is that the point of a logo?

<comment><USER4> The Apple logo says nothing about what "Apple" does but... it looks like an apple.

Likewise, the letters H and P say nothing about Hewlett Packard's business. That wasn't the criticism though. It was that the letters were unrecognizable as an "h" and a "p".

</comment></comment></comment><comment><USER5> I think there's some valid points in this article, but there's also a valid point that kind of undermines a central thesis to this article:

A logo does not need to tell you what it's for if your first viewing is out of context. In fact, 99% of good logos won't accomplish that task. The point is to be recognizable once you know what it is already, and be pleasant to look at over and over.

Now, whether any of the minimalist logos meet that criteria is up to the user. But judging logos on whether you can tell what they're for out of context is a false criteria.

How would you know the Apple logo is for a computer company out of context? You'd be more apt to guess it was for an orchard or cider company. How would you know the modern Windows logo is for the operating system out of context, and not just a bunch of boxes for a moving company or something.

Ambiguity in logos is not remotely a bad thing. What makes a good logo or bad logo is mostly subjective.

The main objective thing I would damn most of the logos highlighted in the article on is being so derivative that they become tedious.

<comment><USER6> > What makes a good logo or bad logo is mostly subjective.

But not entirely. There are some choices that can inherently weaken a design. A logo should be scalable; it should look just as good on a business card as it does on the side of a plane. A color logo should be able to be translated to black & white.

<comment><USER7> True. But most of those rules can be taught over a single lecture. Why then the need for expensive designers? I suspect most of the philosophy around logo design is a simple job-security measure, whether they admit it to themselves or not. For a logo, a person with some sense of aesthetics and knowledge of aforementioned rules is more than sufficient.

</comment></comment></comment><comment><USER8> A well-designed logo relies on its inherent memorability and the strength of its association with its parent brand/entity to create a lasting impression.

It might be informed by the name of the company, but certainly doesn't need to achieve a one-to-one association with the visual letterforms--we don't expect the reverse after all.

Memorability and distinctiveness is the end goal, and if your brand is strong enough that a simple mark will be sufficiently distinctive, then it's to your advantage to adopt a simple mark.

No one will forget that Microsoft is four boxes, and that the Internet is a blue e, even though these marks are visually panned, and in a context-free environment, only loosely associated with their parent brands.

</comment><comment><USER9> This is basically art and design critique by a pedant.

 _" B-b-but their actions don't match their words"_ is the most uninspired of all criticisms. A logo is successful if it generates results, not if it is consistent with whatever principles the critic determines a good logo should adhere to. "You Could Do Almost Anything" is basically the complaint of either unrecognized talent, or the untalented, against the institution for failing to recognize someone whose work follows all the rules, but lacks spirit.

That's what this critique is -- a failure or refusal to see the spirit of the design, and insisting to measure it on a basis that only the author cares for. FFS, the guy drew a bookshelf so he could challenge the designer's intent to portray books on a shelf. Does he think the designer didn't know that a literal application of that principle would result in a broken shelf?

> _One should note that there was a time during which HP intentionally made a > reversible logo._

Should one note this?

This, for me, betrays the true roots of this critique -- unexamined appreciation for all things old, and refusal to recognize merit in new interpretations or to, even temporarily, suspend judgment. Essentially, he is holding their relative newness against them, and revering the old because he is already familiar with those rules. As if those older elements were not once new, and the designers were not also subject to similar critiques from their contemporaries, based almost solely on the fact that new things do not much resemble old things.

The author's own logo fails many of the standards that he holds for these other logos. Apart from giving the impression of inconsistent widths, owing to chamfered interior corners: the "S" of ES would never be unambiguously scrutable, and this logo would always need to appear beside the author's name in order to make sense.

However, despite the fact that his name needs to appear beside the logo, I cannot think of a more incongruous font to accompany that style than a modern sans serif font. The logo is done in the style of an 80s-90s sci-fi/futuristic movie title, and the name is presented in a mid-century sans serif font. I wonder what the author would think of it if he saw it presented elsewhere.

<comment><AUTHOR> > This is basically art and design critique by a pedant. "B-b-but their > actions don't match their words" is the most uninspired of all criticisms.

You're required to be pedantic if you want to examine something with any rigor. And yes, I'm asking that the designers provide justifications that logically cohere with the logo they are selling.

> A logo is successful if it generates results, not if it is consistent with > whatever principles the critic determines a good logo should adhere to.

This is weak. It's a sort of argument to immensity–that if a company is big enough, it can "almost do anything" at all. It's the same argument that Bierut made, and I don't buy it. It's an essentially anti-critical position. One cannot judge a logo by a large company because by sheer weight, they will make that logo stick.

Moreover, I'm not providing the criteria for judgment–I'm pulling them straight from Cooper's mouth.

> is basically the complaint of either unrecognized talent, or the untalented, > against the institution for failing to recognize someone whose work follows > all the rules, but lacks spirit.

Tu quoque fallacy.

> This, for me, betrays the true roots of this critique -- unexamined > appreciation for all things old, and refusal to recognize merit in new > interpretations or to, even temporarily, suspend judgment.

This is invalid on its face. Half this essay is devoted to critiquing a logo from 1964.

> The author's own logo fails many of the standards that he holds for these > other logos.

Tu quoque again.

<comment><USER10> w/r/t the pedantic point, it seems you're a bit stuck in the prison of the literal here. Sure, the books couldn't sit on a real actual shelf, but do they really need to to be understood as books? I got that they were books before I read anything about it and it was a nice little moment. The current configuration allows for three separate moments of understanding: recognizing the symbol, seeing the letters, and then seeing the books. Ultimately, that seems to me like a more rewarding and memorable experience than the letters MITP + color + typeface (no disrespect, Mr. Spiekermann). MIT press's audience isn't stupid, so in my personal opinion hiding a little puzzle in the logo is a nice touch.

I mostly agree with you in your response to the second point. I don't personally find as much fault with the readability issues because the logo would pretty much always be seen in the context required to make it decipherable, but I can understand why one might take issue with that. Ultimately, its important to remember that the logo is neither the brand nor a stand alone work of art. It is only a part of a system.

I think ultimately the pitfall of the don't make me think school of design is that it is susceptible to joyless, by the numbers design but I understand its utility. Also, ironically, one could probably say the same thing about modernist design.

Your article brings up some interesting points and valid criticisms, but I just don't personally agree with all of your values when it comes to design.

<comment><AUTHOR> > Sure, the books couldn't sit on a real actual shelf, but do they really need > to to be understood as books?

If Cooper is to be believed, then yes, we should.

> The current configuration allows for three separate moments of > understanding: recognizing the symbol, seeing the letters, and then seeing > the books.

If there had been no pretense to the second two elements being recognizable, then it would be no problem. Abstract symbols are more than acceptable. The issue arises when designers invent a story about a representation which is hardly represented at all. Until it is explained, most readers will not get to step two or three.

> Ultimately, its important to remember that the logo is neither the brand nor > a stand alone work of art. It is only a part of a system.

True, and yet it's still important to design the logo intentionally for when there is no contextual system. If the logo is to be abstract, great. If not, that's also great. But no one should fool themselves about where they are on that spectrum.

> I think ultimately the pitfall of the don't make me think school of design > is that it is susceptible to joyless, by the numbers design but I understand > its utility. Also, ironically, one could probably say the same thing about > modernist design.

Absolutely. I don't count myself as a member of the former school. I assert designers should merely be honest about their intentions.

> Your article brings up some interesting points and valid criticisms, but I > just don't personally agree with all of your values when it comes to design.

That's the fun of it!

<comment><USER10> Whoops, there was a typo in my post. I meant to write do they need that (meaning a physically accurate bookshelf) to be understood as books.

>That's the fun of it!

No, I am the arbiter of design and all who disagree with me are evil and wrong! Who moved my crown??

</comment></comment></comment></comment></comment><comment><USER11> IEEE's Spectrum changed to a Cooperesque logo a few months back. I used to like it, but after reading this article, I'm not so sure.

[http://spectrum.ieee.org/](http://spectrum.ieee.org/)

"consistency is more important than cleverness. Consistency is actually really hard to achieve. Cleverness is a cheap commodity."

I hope I can pull a Cooper the next time I am accused of being derivative and lazy.

<comment><USER12> As TheOtherHobbes notes elsewhere in this thread, your first impression may have been fair. The Spectrum logo is - at least to me - certainly reminiscent of spectral bands stripped to the minimum idea. Likewise, a technologist is likely to see the MITP logo and grok the various visual puns linked to barcodes, books, graphs, tape, 1s/0s, gear teeth, precision, and all manner of other tech thingies. What _exactly_ each person sees may be different, but it sure feels appropriate.

(Side note: I feel that having to stare and figure out the logo really accentuates that MIT hacker mentality. It's a small, clever joke or puzzle, and that feeling of "oh I get it" seems superbly appropriate here. That confusion? That sort of exclusive humor/cleverness is classic hacker in-joke material.)

</comment></comment><comment><USER13> This reminds me of the "SASV" laptop for sale screenshot (ASUS).

</comment><comment><USER14> Side note: The new minimalist HP logo looks like the outline of someone's hand flipping you the bird. Once you see it, you will never un-see it.

</comment><comment><USER15> Is there an inshort for windows?

</comment><comment><USER15> IS There a Inshort for windows?</comment>


A cool way to explain logic gates with water [video] - <AUTHOR> https://twitter.com/page_eco/status/1188749430020698112 <comment><USER2> I suppose you could call this "dynamic water logic" because it relies on flow instead of pressure. I was expecting hydraulic relay valves ("static water logic"), which are widespread in industrial control applications, and perhaps even more commonly, the automotive automatic transmission:

[http://www.cogpro.com/chapters/E-CruiseOMatic/images/E-Cruis...](http://www.cogpro.com/chapters/E-CruiseOMatic/images/E-CruiseOmatic0051_jpg.jpg)

Almost all the material I could find about computing with water seems to focus on dynamic gates, essentially completely ignoring the century-old[1] knowledge of hydraulics that's been present all along. Perhaps this shows just how isolated the different disciplines can be.

[1] I have a book from the early 1900s about water-operated elevators, complete with descriptions of all the valving required to control the hydraulic ram. They were logic gates, before they were even called logic gates!

<comment><USER3> The book The Three Body Problem also has a great scene explaining logic and computational math via ... traditional means :-)

<comment><USER4> I was thinking the same thing! Great book, highly recommend

</comment></comment></comment><comment><USER5> I want to build a megalithic computer using a river flowing down a mountain. The AND gate is a enormous door lifted by an enormous arm attached to an enormous block hollowed out from the bottom so that it can float.

Input is done by almost lifted doors that can be moved by applying enough man power.

The display is just enormous blocks lifted up.

It should do something important like play pong.

</comment><comment><USER6> Now someone should take this to the next level and makes a fountain that implements an 8-bit adder.

<comment><USER7> Will 3 bits do?

[https://www.youtube.com/watch?v=6qP9HfUOCN4](https://www.youtube.com/watch?v=6qP9HfUOCN4)

<comment><USER8> Is that a text-to-speech voice?

<comment><USER9> Yeah, lots of science videos do this, particularly when the authors are non- English-native and (presumably) nervous about their voice/accent.

<comment><USER8> It's one of the more interesting TTS voices I've heard, particularly in terms of where it places emphasis. Any idea what it's named?

</comment></comment></comment></comment></comment><comment><USER10> i don't think this would 'explain' a logic gate to anyone who doesn't already understand a logic gate. this model doesn't convey any information that would help someone reason about a logic gate. If you didn't know anything about logic gates and you saw this, how would it improve your intuition? If I imagine these water streams in my head, does it help me understand how, why or what logic gates do?

i learned logic gates playing mods for games I was playing... I could have a motion sensor and see when I NOT'd it it made its signal the opposite. I could AND two of my sensors and see that I now had a signal combining both. I think this kind of thing, a model that simulates _how_ logic gates are used gives a much better intuitive understanding of what they're for and how to use them.

<comment><USER11> It _totally_ worked for me.

I have zero physics background, and this is something I've never understand. I knew what a logic gate _did_ , but the core idea of "how does a gate just _know_ whether it's _this_ AND _this_ " always seemed like magic.

I had a serious lightbulb moment with this video. Although, I had to download it and slowly scrub through several times.

I still don't actually understand how this would work with electricity, but I can see how it's broadly possible.

<comment><USER12> This doesn't help you see how it works with electricity, because it's not really how it's done with electricity.

Put two valves after each other on a pipe. The water can only flow if the first valve AND the second valve are open.

Fork off a pipe into two pipes with a tee, put a valve on each side, and connect the other ends of the valves together with another tee. Water flows through the whole thing if the first valve OR the second valve is open (or both).

That's how it works with electricity too. The valves are themselves controlled with water/electricity too.

<comment><USER2> Yes, series and parallel connections are the underlying theory behind electrical logic gates, and work with plumbing too.

I believe something like this is also discussed in Petzold's famous book ([https://en.wikipedia.org/wiki/Code:_The_Hidden_Language_of_C...](https://en.wikipedia.org/wiki/Code:_The_Hidden_Language_of_Computer_Hardware_and_Software) )

</comment></comment></comment></comment><comment><USER13> I strongly suspect Ted Stevens was thinking of hydraulic systems and logic when he made the "series of tubes" comment.

He attended engineering school in 1942, around the time that hydraulic systems underwent a resurgence.

</comment><comment><USER14> I know HN is normally above such low brow things, but since these were made by the original creator, I think it's okay...

Logic Gates with "Yellow Water": [https://gfycat.com/radiantimportantfieldmouse](https://gfycat.com/radiantimportantfieldmouse)

Logic Gates with Slurm: [https://i.imgur.com/UJyNd9T.gifv](https://i.imgur.com/UJyNd9T.gifv)

</comment><comment><USER15> So, water is turing complete.

<comment><USER16> No. No more than electricity is turing complete.

</comment></comment><comment><USER17> cool. now if someone could just explain water flow with logic gates...

</comment><comment><USER18> I kind of like the way the Ontario Science Centre taught logic gates which was by using balls on tracks rather than water.

</comment><comment><USER19> If the navier stokes equations are turing equivalent there's no analytic solution.

</comment><comment><USER20> A typical example that just appears to be "insightful" while what is happening in the actual logic gates has no similarity with that at all:

[https://www.youtube.com/watch?v=58nSgVvXVbo](https://www.youtube.com/watch?v=58nSgVvXVbo)

so any "intuition" one might hope to gain from these two pipes will be wrong.

Therefore "explain logic gates" is an incorrect title. It can be used to explain the truth table for AND, but not the gates.

<comment><AUTHOR> "Logic gates" is an abstract concept, and yes, it is essentially a truth table.

You refer to one of the transistor-based implementations of logic gates. Yes, it works differently.

<comment><USER20> I refer to any kind of logic gates actually used in the devices we use -- otherwise what is there to "explain"?

<comment><USER21> The boolean algebra part.

<comment><USER20> For "the boolean algebra part" the table like from here (second slide) is a real _explanation_ :

[https://courses.cs.washington.edu/courses/cse370/03au/lectur...](https://courses.cs.washington.edu/courses/cse370/03au/lectures/02-Comb.pdf)

"There are 16 possible functions of 2 input variables: in general, there are 2^(2^n) functions of n inputs"

"And" from the original tweet is just one of them.</comment></comment></comment></comment></comment>


The Bitcoin Model for Crowdfunding - <AUTHOR> http://startupboy.com/2014/03/09/the-bitcoin-model-for-crowdfunding/ <comment><USER2> I'm pretty sure giving investors worthless virtual coins instead of actual stock certificates is illegal.</comment>


Developer salary growth is an inverted hockey stick - <AUTHOR> http://blog.hirelite.com/developer-salary-growth-is-an-inverted-hockey

<comment><USER2> Five years of experience as a programmer is like becoming a "made man" in a crime family. Before, you're nothing. After, you're trusted and respected.

I was in the same job from years 2-5 and didn't see much of a raise. Then when I got a new job it was more than double the salary. And it does seem like you have to leave to go somewhere else, or else you're a prophet without honor.

It's also clear from the graph that after 5-10 years of experience, much of the increase in income growth will be expected to come from non-salary compensation. Which may be equity, bonuses or stock plans, but it's all about getting a piece of the action. Which, again, makes programming sound like it's the mob. Sorry about that.

<comment><AUTHOR> Was there anything you did as far as negotiating the salary at the new job or did they offer you that much straight up?

<comment><USER2> This was years ago but I mention it since the demand for developers was similar.

I was going from a really low-paid games job to doing application programming for a large startup. Game programming tends to have even more depressed starting salaries than usual, although this was back when everyone was still salaried.

My former boss actually predicted that I "could make twice as much somewhere else" when I left, and the new employer in fact offered that straight up.

</comment></comment></comment><comment><USER3> In economics the reverse hockey stick growth is known as diminishing marginal returns. This simply means the market values the earliest years of experience as much more beneficial to a developers skill than the later.

While I understand the idea behind hiring young and inexperienced, it should be noted that the wage rates are there for a reason. The training costs/possible managerial time may be greater for the average college graduate. (Clearly not always true- I am just trying to postulate why the market values experience this way).

<comment><AUTHOR> You're right about training costs for college graduates. Another thing to consider which isn't directly mentioned in the post is training across a (programming) language difference. If someone is a C# developer and wants to join a company that uses Ruby or Scala, they might be willing to take a pay cut to get experience with new technologies.

<comment><USER4> That's an excellent point. Personally I've never considered that because I feel that experience 'programming' beats experience 'programming $language', at least for people who are above average.

Above average means giving a shit about being a developer, not just being in it for the employment.

</comment></comment></comment><comment><USER5> This is true in most industries. You either move on to a management position or stagnate. The problem is it doesn't really work in the development world where you can get better and better the more deeply you use and understand the technology. Plus it has the side effect of pushing highly technical people with no management skills out of jobs they're good at and into jobs they can't do well (imagine Woz trying to be a manager)

That's why many big tech companies have moved to a "level" system (to the best of my Knowledge Microsoft started this trend but I could be wrong). In that type of system you can continue to be a developer but move up in level and salary. That way great developers can stay great developers and not be penalized for it.

<comment><USER6> > _That's why many big tech companies have moved to a "level" system_

Although you have to do it right. Netscape had such a system, with levels ranging from T1 to T7. I started at T4. Two years later, I was doing work that, according to the written definitions, should have put me into T5. I asked my manager if I could get promoted; he bumped it up to his director. About four months later, the director answered me: yes, I did meet the written definition, but, no, I couldn't have a promotion, because they didn't actually use the written definitions. They were guidelines, but the real standard for promoting to level N was comparing the person to other people who were already at level N...and there _were_ no T5s in the company.

Basically, it was a firebreak, which provided the appearance of a career path, without providing the reality. (I expect this wasn't on purpose, but that was the effect.) Three months later, I moved to another company, hired as what Netscape would've called a T7.

</comment></comment><comment><USER7> This has created a pretty serious issue at my current job. My experience is increasing in value faster than policy lets my salary appreciate.

Policy is forcing me to find a job elsewhere. I cannot rationalize continually taking a relative pay cut to stay where I am.

</comment><comment><USER8> There is an interesting subtext in this article, which is to suggest that you take _more_ risks as you get closer to the asymptote of salary growth, because it's the way to avoid diminishing returns for the additional skills you've learned between hitting the level of "experienced" and now.

It's the opposite advice that makes up common wisdom, which is to take risks when you are young, inexperienced and without responsibilities, and then settle into the comfortable long-term job with small raises once you get past that phase. I'm not surprised by this; if technical skills compound at _all_ , it would seem that a developer's value would trend exponential and not logarithmic. That is, so long as they don't weary and stagnate.

But then, there is also value for the inexperienced in a startup, in that you'll learn a ton, and be given a lot of responsibility and autonomy. It might be that common wisdom is only half-broken.

Granted, there are opportunity costs involved. You might be in a position where you have to trade capitalistic striving for a paycheck, as a ribbonfarm article posted here a while back put it. It's as much being aware of those opportunity costs, though, as it is knowing whether or not you are in a position to need to accept them.

<comment><USER9> Not quite. I think this is conventional wisdom:

If you take a risk and co-found your own startup with 0 years of experience, then even if you tank 2 years later, you are likely to wind up close to the 5 years of experience point of the hockey stick when applying for a job.

Basically, the startup gives you a small chance of huge upside, and on the downside, you become a normal W-2 salaried employee but begin at a point farther along than someone who played it safe the entire time.

Whether this happens in practice is probably highly dependent on whether the risk taker demonstrated some goodness during the failed startup.

<comment><USER8> _If you take a risk and co-found your own startup with 0 years of experience, then even if you tank 2 years later, you are likely to wind up close to the 5 years of experience point of the hockey stick when applying for a job._

That's probably closer to the conventional wisdom around HN, but it is a far cry from conventional wisdom amongst the masses. The wisdom amongst the masses is that if you co-found or join a startup, you are playing roulette with your financial security.

But the article's point -- which I agree with -- is that even if you accelerate this process of getting to that 5 year threshold, you are doing yourself a disservice fiscally by settling in at the plateau unless you have a good reason forcing you into settling.

You're likely still learning more, and becoming more valuable, but it becomes much more difficult to extract that value through salary. So, it becomes beneficial to join companies where that only makes up a part of the total compensation package.

</comment></comment></comment><comment><USER10> It's more of a 180-degree rotated hockey stick.

<comment><USER11> All hockey sticks are.

<comment><USER10> Only if both ends are considered the same or outside of flatland.

</comment></comment></comment><comment><USER12> Just thinking out loud here.

Could it be that after 5 or so years engineers are in long term relationships, perhaps kids, etc and thus are less inclined to rock the boat and ask for a significant raise.

Could it be that a lot of developers leave after about 5 years or so and go into other fields (management, startups, etc) so that the people left in the job for 10+ years are actually comfortable with just plodding along?

Then again I'm left thinking that it could be because there actually isn't much of a difference between a developer with 5 and one with 10 years of experience (in the general sense).

This is interesting to me, since I've just gone past the whole 5 years of experience thing, hit a limit with one company, and moved elsewhere for nearly 2x the money. Wondering what/where I should go next.

</comment><comment><USER13> It seems to me that this assumes developers never get promoted (which makes the long term look unappetizing). But developers become lead developers, architects, and ( _gasp_ ) managers.

</comment><comment><USER14> How do I get a job at 0 years working remote from Houston?

<comment><USER15> I've been working remotely for 5 years doing all sorts of web development. The two things that have helped the the most are:

1) Open source contributions.

2) Personal projects.

I still haven't made any money from any of those but putting them in the emails I send to offers found on the different job boards have always put me ahead in the line of people applying.

If you're desperate for money you can always go to sites like oDesk and applying for stuff there. You'll have to accept $15-$30 per hour (or even lower when starting) but once you get a good reputation you can make a decent salary out of that.

Finally, write me to firstname.lastname@example.com if you have a semi-decent Github profile and I'm sure we can reach an arrangement!

</comment></comment><comment><USER16> A labeled y axis or error bars would be nice...just to dispel that vague link- spammy feeling.

<comment><AUTHOR> The y axis doesn't have specific units because they vary so much by location. I tried to mention a few specific data points below the graph to provide a bit more context. Good point about the error bars. I'll try to include them in an update to the post.

<comment><USER16> any chance you'd release the raw data? my email is in my profile, i'd be happy to share the results.

</comment></comment></comment><comment><USER17> I'd probably call it a logarithmic function where dying is a problem in more than just a "boundary value" sort of a way ;)

</comment><comment><USER18> This got blocked by the proxy server at work. Can anyone summarize (if a summary is even needed beyond the title)?

<comment><USER19> I think they're telling you to find a new place to work.

<comment><USER6> Speculation: maybe the proxy server blocked it because the domain name includes the word "hire", and the company is trying to _keep_ people from finding a new place to work.

<comment><USER20> This Websense category is filtered: Blogs and Personal Sites.

<comment><USER21> Though that could be a lie to make you not know they are blocking you from looking at 'hire' websites.

<comment><USER19> Either way, I find much of the best information on some narrow topics are on sites that could be described as "blogs".

</comment></comment></comment></comment></comment></comment><comment><USER22> For a moment I feared to see that chart the other way round (starting high, ending low).

</comment><comment><USER23> Anyone have a sense of how true this is at big places like Google and Microsoft?

<comment><USER24> One thing not taken into account in this discussion is the natural limitation of career growth. Everyone has a cap on their advancement. Some devs hit that cap after 4 years, some after ten. But a smaller and smaller percentage of employees are promoted to a higher level after the requisite amount of experience.

At Google and Microsoft there is very large percentage (~60%)of employees in the T4/SDEII bracket. This is the 2-5 year experience bracket. The percentage of employees in a level exponentially tapers off when compared against higher levels. This could be one explanation of the graph as presented.

At the companies you mention, the graph is accurate when plotted against population but incorrect when plotted against level.

Mid to higher levels see approx linear growth in salary and compensation until the top echelon. The compensation difference between two levels is approximately 10% the starting salary of a college hire. However, this highest levels including directors, tech managers and partners see clearly non- continuous bump in salary and equity (2-5x salary, 10-50x stock).

</comment></comment><comment><USER25> I'm surprised nobody has discussed the selection bias going on here. The better programmers are the ones that will stay on. IDK what the general turnover is in coding, but it makes sense that some people won't be able to hack it, appearing to boost the salaries of the "better" workers.</comment>


'Spam king' Wallace indicted for Facebook spam - <AUTHOR> http://news.cnet.com/8301-1023_3-20088487-93/spam-king-wallace-indicted-for-facebook-spam/

<comment><USER2> Same story:

<http://news.ycombinator.com/item?id=2849064>

<http://news.ycombinator.com/item?id=2849438>

<http://news.ycombinator.com/item?id=2849570>

<http://news.ycombinator.com/item?id=2850338>

Currently none of the HN submissions have comments worth reading.</comment>