The violence paradox: Is violence actually declining? - <AUTHOR> https://www.pbs.org/wgbh/nova/video/the-violence-paradox/ <comment><USER2> This is long, and I suspect most of us have neither watched it nor read all the transcript; but from the parts of the transcript I have skimmed it seems to hit absolutely all the areas I would expect it to cover. Everything from evolutionary biology to memetics through colonialism, inequality, the epidemiology model for interventions, sports across cultural divides. I suspect most sentences are themselves summaries of a stack of investigations.

Stephen Pinker may get some flak for being a pop-science writer but this seems extremely thorough. It would be nice not to use this article as a platform for exercising our pre-existing prejudices.

</comment><comment><USER3> Violence as a contagious disease is a concept that isn't going to fare well in certain circles (of proponents of open borders, migration etc.). More research should be done in this area, if possible.

<comment><USER2> A border is a structure that requires violence to maintain.

<comment><USER4> Well if we want to be like that, then everything requires violence to maintain.

The Empire State Building takes violence to maintain because without the threat of violence someone might carry away all the rocks. Your local library takes violence to maintain because without the threat of violence someone might burn all the books. Your own life requires violence to maintain, because only the threat of violence from the state prevents someone from killing and eating you.

<comment><USER5> That is basically just a snarky way of restating the premise of the monopoly on violence[0].

And we do want to be like that, in theory, because violence cannot be avoided entirely in human interactions, and because violence as a regulated market is more humane than violence as a free market, and because that violence in the former case is mostly abstracted into the bureaucracy. For many people in a modern society the closest they'll ever come to seeing the "violence inherent to the system" is maybe a speeding ticket or misdemeanor jail time.

[0][https://en.wikipedia.org/wiki/Monopoly_on_violence](https://en.wikipedia.org/wiki/Monopoly_on_violence)

<comment><USER3> > _And we do want to be like that, in theory_

One problem is that the whole illusion of power a government requires to operate falls apart when this promise isn't kept in practice and a large number of people become aware of this failure through media. Perhaps the news and following public outcries about e.g. police violence in the US serve a (second) purpose.

<comment><USER5> Yes, in exchange for the people ceding their right to violence to the state, states are supposed to be trustworthy and neutral arbiters who follow their own rules and are subject to the will of the people. Both the people and the state are supposed to be limited in their power.

When that falls apart, and people decide to no longer participate in the fiction that is civil society, things can get ugly.

</comment></comment></comment></comment></comment></comment><comment><USER6> > Is violence actually declining?

We know it is. Not in question at all. But we don't know for sure if war is declining since it's overwhelmed in large spikes.

Good doco.

I liked the theory good manners is one possible reason. It's cute but actually makes sense, although not a huge fan of the it works part of the brain more.

I'd go with a lot of violence is embarrassing and we are taught embarrassing is bad. Sit up straight, use a knife and fork and don't hurt people.

</comment><comment><USER7> I can't view the video but since Pinker is mentioned in other comments I assume it is a summary of this (not yet read) book on my bookshelf:

[https://en.wikipedia.org/wiki/The_Better_Angels_of_Our_Natur...](https://en.wikipedia.org/wiki/The_Better_Angels_of_Our_Nature)

</comment><comment><USER8> Here in the Netherlands, crime statistics are declining, but many people suspect that the real reason is that the police often refuses to register declarations (or advises against filing declarations).

<comment><USER9> Does the Netherlands not have the equivalent of [https://crimesurvey.co.uk/](https://crimesurvey.co.uk/)?

<comment><USER10> I bet the dutch central office of statistics keeps track of it. But then if you don't trust the one government body why should you trust another?

Personally I trust the dutch police force and in fact admire them greatly. They handle dangerous and stressful situations much better than I personally ever could. That said I do see why you'd be mistrustful as average citizen, the most you'll have to do with police is reporting crimes such as broken windows or stolen bikes, in which case you'll probably get a shrug and a "we'll look into it" at most.

The police force is currently _very_ well trained and adequately supplied, but as is the story in many places budget cuts are being made all over and I worry for the future.

In any case an independent research office looking into the crime stats wouldn't be a bad thing, but I suspect it's a rather tricky thing to gauge in the first place.

<comment> the-dude Police stations are closed en masse. Police has been in reorganisation since forever. Personnel is over-extended to the point they are contemplating strikes.

<comment><USER10> > Police stations are closed en masse.

I can't find any info on that for the dutch police, do you have a source?

> Police has been in reorganisation since forever. Personnel is over-extended > to the point they are contemplating strikes.

Yep. And it's getting worse. For now it's all still operating well enough but we're at an inflection point for sure.

<comment> the-dude [https://www.rtlz.nl/algemeen/binnenland/artikel/4394841/poli...](https://www.rtlz.nl/algemeen/binnenland/artikel/4394841/politiebureau- verdwijnt-uit-het-straatbeeld-bijna-200-bureaus)

Are you actually living in NL?

<comment><USER10> Yes I do live in The Netherlands. Excuse me for not remembering a year old news item that didn't have any direct impact on me.

Thanks for the source, closing police stations is definitely not a good sign. I hope that that's a trend that has reversed since that news article came out.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> Violence is changing by getting a more peaceful face in order to be "legalized" and "digested". Violence is always here and it gets stronger.

<comment><USER12> What do you mean?

<comment><USER13> I don't know what the parent means however isn't it true that some cities in the US stopped prosecuting some crimes deemed "petty" such as theft, "hard drugs" dealing, …

Which would make crime statistics lower than they really are.

<comment><USER12> I think "under reporting crime" is not something very recent. But something that is usually not under-reported, is violent crimes like murder or assault resulting in severe injuries, because someone ends up at the morgue or in the hospital where they are definitely tracked and recorded. As far as I know this is where we are pretty much sure violence is decreasing over time.

</comment></comment></comment></comment><comment><USER14> The world goes through cycles of peace and violence.

We are due for the next cycle of violence to rival WW2, if not even more so. We have forgotten and are unable to imagine when more than a hundred thousand are killed in a single day. But those days will come again and very soon.

The next war will be (pretty much) the West against the Rest. The whole of the West has a smaller population and less manufacturing capacity than China alone.

<comment><USER15> If you haven't noticed by now, the world is a plutocracy/oligarchy.

It hasn't been about nations for a long time, but the rich and powerful people who influence them. Many of them aren't even elected officials.

See Trump and Russia, Saudi Arabia and Khashoggi, the American citizens beaten up by Erdogan's bodyguards on American soil, the UAE and Sheikha Latifa, American corporations kowtowing to China, the US ambassador's wife who got away with murdering a UK teen, Jeffrey Epstein and his private island of underage prostitution for the cream of society's crop, and the _many_ other cases where the personal whims of the elite few override sovereign laws.

Those people literally have access to everything that human civilization has to offer, and they're not going to ruin their playground.

<comment><USER16> Hasn't it always been like this though? At least nowadays they don't claim to just be better humans (nobility).

<comment><USER15> Indeed, the ruling class never stopped ruling.

But this is the first time in human history, when they can go anywhere, do anything, to anyone, at anytime, without having to worry about national borders etc.

Money is the universal differentiator now, and anybody of any nation or race can have it.

Until you piss off someone else who also has money, then connections come into play, _then_ the might of arms.

<comment><USER17> This comment seems strained. A history book would neither back up the claim about unfettered travel, nor the claim about corruption.

The parent comment actually seems reasonable to me. It really wouldn't take much escalation for a trade war to turn into a shooting war between, at least China and America.

<comment><USER15> > A history book would neither back up the claim about unfettered travel, nor > the claim about corruption.

I don't think any ruler would have been able to easily hop to another continent on a whim for an orgy with a wide selection of sex slaves and be back home by next dawn, before the advent of instantaneous communications and jet planes.

The point was that the people who command nations have access to a previously unprecedented level of luxury which would be ruined if the world wasn't largely at peace.

<comment><USER17> Ah, okay. I thought you were referring to well-defined borders, and strict border controls. Both these were generally lax before the 20th Century.

I don't see how this poses much of a safeguard, in any event. If millions of Chinese wind up starving due to a trade war, for example, a shooting war is almost inevitable.

<comment><USER18> > I don't see how this poses much of a safeguard, in any event. If millions of > Chinese wind up starving due to a trade war, for example, a shooting war is > almost inevitable.

It doesn't.

A shooting war has already commenced by the CCP in Hong Kong because they refuse to acquiesce to the dominion of the PRC's illegal dictates, so what you should be worried more about creating a 'shooting war' is China's economic turmoil (much by the machinations of the CCP/PRC) having a spill-on effect outside of its borders than within. Moreover, a correction is long over-due, just to be clear, the US shouldn't be a super power anymore, either; nor should any nation if we've seen what they do with said power.

While I agree the World is under a kleptocractic-plutacracy, the emergence of the Nation-state was only ever just that; there was no 'Golden era' of the State, it was only forced down the serf/peasant class' collective throats and legitimized under a thinly-veil scam called an election process which promotes division and mob-justice. One in which can be altered/manipulated in masse and in certain cases outright removed as it sees fit.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER19> Nylonkong are getting more attention with Trump, Brexit, and the protests. When was the last time you read an in-depth news about some under represented region, say, Nigeria? For a country experiencing population boom and worsening internal conflicts, 200 million people there ARE going to fight for fresh water and other resources if the rest of world just sit there and do nothing.

<comment><USER20> What is 'the rest of the world' supposed to do? It's not like they don't have access to internet or condoms (even free), not in 2019. It's a (collective) choice. See [1]. It's about Niger (same largest ethnic group - Hausa[2]), which is also experiencing a population explosion, but the principle appears to be the same:

>"I just pray to God to bless those three babies I have," she says. The local health centre in her village of Darey Maliki offered her free contraception, which they get partly from the NGO Pathfinder, but Hamani declined. "Maybe [my in-laws] would tell my husband to marry another woman to have more babies," she says. "If they want me to have another pregnancy, I can do it just for them to feel happy."

Perhaps that culture changes in the future generations, but for now Subsaharan Africa is on track for 2B+ in 2050, and nothing short of starvation or violence can change that.

[1] [https://www.theguardian.com/global-development- professionals...](https://www.theguardian.com/global-development- professionals-network/2017/mar/15/why-have-four-children-when-you-could-have- seven-contraception-niger)

[2] [https://en.wikipedia.org/wiki/Hausa_people](https://en.wikipedia.org/wiki/Hausa_people)

<comment><USER21> Where infant mortality is high, humans tend to have more children.</comment></comment></comment>


Chrome Extension Replaces Every Instance of 'The Cloud' With 'My Butt' - <AUTHOR> http://betabeat.com/2013/03/chrome-extension-replaces-every-instance-of-the-cloud-with-the-far-superior-my-butt/

<comment><USER2> <http://cl.ly/image/0M2g1Z1G3G0M></comment>


How humans kill companies - <AUTHOR> http://hitenism.com/how-humans-kill-companies/ <comment><USER2> Way too involved in your team's day-to-day- I'd say that's a pretty big issue. When you hire a designer to design, and you're telling them where to place things and what they should look like when you yourself have no background in design, that's a sure-fire way to bring a company down

</comment><comment><USER3> The problem is that office politics are usually the only way to resolve issues within a company.

Invariably there's someone tasked with deciding which language something will be developed in, whether you can develop something in a given language, etc.

Usually these things are decided based upon whether the stupidest programmer in the room can program in this language, and whether in general bottom of the barrel programmers can use that language. Whenever you see Java, PHP, or C# you'll usually find the reason has something to do with the ability to hire 200 programmers to do the work of 2 or 3.

This is the office politics that kill companies, the usual Jimmy doesn't like Frankie stuff is minor and easily dealt with. Also, it generally only actually kills the company once a competitor is operating inside their OODA loop. In general with out a competitor that doesn't have office politics there's no serious threat to the organization.

<comment><USER4> > Whenever you see Java, PHP, or C# you'll usually find the reason has > something to do with the ability to hire 200 programmers to do the work of 2 > or 3.

You can do a lot better than that. Java, PHP and C# are sometimes simply the right tool for the job, and _no_ programming language gives you a 100:1 advantage. That's simply ridiculous.

<comment><USER3> The right tool in the right hands will certainly give you that kind of advantage.

It's not in language alone, but combined with the craftsman, the level of trust in the team, and the process, certainly.

Once you have 200 programmers you have to have managers, architects, etc, and you have communication overhead. I'd say off the top your communication overhead is 50%, so lets say you now have 100 programmers available to program, and 100 to communicate. Now to use these 100 programmers effectively you need them to work in parallel, in any project you usually have layers so some programmers simply won't be able to do anything because they are waiting for someone else, lets say outside of meetings these programmers are actually programming 75% of the time, now you have 75 programmers. Lets say java is twice as inefficient in terms of LOC as ruby, now you have 35 programmers, lets say a ruby rockstar is 10X what the average java programmer is, now you have 3.5 programmers.

When you walk into your typical large corporate dev shop you can take on many times the workforce with a small team of dedicated craftsmen who trust each other.

Whether it's 50:1, 100:1, or 200:1 I'm not sure, but I am sure that it's in that ballpark.

<comment><USER4> I challenge you to write an application like Wowza in ruby.

Java is insanely performant for some workloads, as a long time C hacker I think that is interesting because this used to be an area where C was unchallenged and now java has achieved parity there, and in some situations can exceed C. Ruby and other glorified scripting languages need not apply.

LOC is not a useful measure for productivity anyway, I thought we had put that to rest a while ago. What matters is whether or not the stack you are using gives you what you need, if what you need is fewer lines of code because you've made that a metric then you are missing the wood for the trees.

Long term what matters is maintainability, control of the development process (which you should be able to achieve in any environment but which appears to be problematic in some), your ability to find employees and so on. Lines-of- code never was a business deliverable and it should not be, it does translate into some measure of cost in the longer term but that cost can be offset by lots of factors.

</comment></comment></comment></comment><comment><USER5> Black sheep do exists everywhere even if you find them and fire them, they go out and create rumors about the company.

I've faced this twice.</comment>


Rechargeable Batteries with Flexible Solar Cells - DIY prototypes - <AUTHOR> http://blog.bareknut.no/2009/02/rechargeable-batteries-with-solar-cells.html

<comment><USER2> This is a great idea. Why hasn't any one done this before?

<comment><USER3> I'm guessing it's because exposing batteries to direct sunlight isn't a safe thing to do. Additionally, it might reduce the recharge capacity by affecting the components inside.

</comment></comment><comment><USER4> They would be very useful in survival situations.</comment>


Peer-to-Peer Communication Across Network Address Translators (2005) - <AUTHOR> http://www.brynosaurus.com/pub/net/p2pnat/

<comment><USER2> So basically here I am, opening a connection to the publicly accessible S. The router sees it and says _oh wow, there 's an out going connection to S_IP:S_PORT, I'll assign an R_PORT to it on my publicly visible side and will forward all traffic from S_IP:S_PORT to my internal network client_.

But how does that help someone who is not S (has a different IP than S) to use this tunnel set up by router? It (R - router on the NAT) won't ignore the source IP of incoming packets (since two connections are the same only if source and destination IPs and ports are the same - 4 numbers), will it? Traffic coming from S will get forwarded, that I get, but will traffic from ANY source just coming to the appropriate port on R get forwarded just the same way?

Good paper.

<comment><USER3> You are right but your logic works in tcp connections, that why the method named "udp hole punching", in udp there is no handshake that means i can send you a udp packet looks like it's coming from S_IP:S_PORT.

</comment></comment><comment><USER4> It's a good paper for anyone starting up with the NAT traversal, but it missed few things even when it first came out, so just make sure to read more on the subject if you're actually implementing the UDP/TCP punching.

</comment><comment><USER5> I spent 2 years on a cross platform p2p protocol with features borrowed from GameCom, RakNet, Zoidcom and the Torque Network Library (TNL). I failed, largely due to the complexity of overcoming NAT. This was before ZeroMQ and other libs made it easy to emulate TCP traffic on top of UDP (which IMHO is the way it should have worked in the first place).

What concerns me most is that IPv6 supports NAT and so I have a feeling that we will be living with it for decades to come (effectively making large swaths of the population second class citizens, as they will perceive the internet as download-only).

Does anyone know of a library that universally overcomes NAT, using STUN/UPnP/NAT-PMP and other techniques where necessary? Ideally it would work by handling the handshake and return a standard UDP port that "just works", giving the app a globally accessible public port.</comment>


Former CIA Chief of Disguise Explains How Spies Use Disguises [video] - <AUTHOR> https://m.youtube.com/watch?v=JASUsVY5YJ8 <comment><USER2> With Facebook and facial recognition, spies can be identified _before they even know they want to become a spy_. There is literally a lifetime of facial recognition data on social networks for youths and college grads as they post their pics online, and if they get recruited to become a spy in their 20s, there will already be enough information to identify people globally, as they walk through airports, shopping centers, etc and have their pictures videoed constantly.

Facebook, et. al is a treasure trove of data for purposes like these.

<comment><USER3> I would assume that in front of the headquarter of any major intelligence agency, there is a dozen minivans, one for each foreign agency, taking continuously recording+facial recognition of employees coming in an out. Useful to spot foreign agents when they come through your airports security.

<comment><USER4> One would imagine SAD/SOG types are not exactly walking in through front door there are prob protocols in place for this type of things.

<comment><USER4> Unless we are talking about something like Russian GRU than they will basically list GRU building as their registered address in various open and semi open databases.

<comment><USER5> What are you talking about? Active GRU agents don't get designated as GRU and get embedded in other units.

Even on the direct action level, they aren't listed as GRU anywhere, wear the uniform of some regular unit in the area, and have paperwork saying they are a regular soldier.

<comment><USER6> Referring to [https://www.bellingcat.com/news/2018/10/04/305-car- registrat...](https://www.bellingcat.com/news/2018/10/04/305-car- registrations-may-point-massive-gru-security-breach/)

Which is kind of a feel good story, point and laugh at those Russian idiots bungling things again.

<comment><USER5> Cool, thanks for the link.

</comment></comment></comment></comment></comment></comment></comment><comment><USER7> I wonder how good the disguise "technology" is that they won't disclose in a public YouTube video.

I don't know why you'd want to disguise yourself as the opposite gender unless you could pull off the voice, your cover would be blown almost immediately.

I still don't place any stock in intelligence operatives working in embassies, I can only imagine it works because intelligence agencies aren't sophisticated enough yet. You have a (relatively) small group of staff, and you _know_ a portion of them are spies. Many "official covers" are extremely weak.

<comment><USER8> Silicone prosthetics are nearly perfect disguises.

And if you pull off a female illusion well, youd be surprised that most people wouldnt think twice about your voice.

<comment><USER9> The speaker in the video claims that they can turn a woman into a man but that it's nearly impossible to turn a man into a woman. In other words, according to her, there is no way to "pull off a female illusion well".

<comment><USER8> _Nearly_ impossible, sure. But not impossible. There are professional female illusionists you can look up to see the amount of preparation and effort it takes to pull off.

</comment></comment></comment></comment><comment><USER10> I'm wondering now when all these classic behaviors become imprinted. I'm born in America, grew up in Germany for the most part. Apparently I eat and stand like an American.

<comment><USER11> Another interesting one people don't notice is counting money.

[https://www.youtube.com/watch?v=g87HVlu55mQ](https://www.youtube.com/watch?v=g87HVlu55mQ)

I saw this video and didn't think much of it until I saw my friend counting money the Asian way. He didn't know that wasn't how Americans typically did it - I guess you pick up a lot of little things like this by watching your parents.

<comment><USER12> I felt the same when moving to Romania, they have a different way of counting money compared to France. I barely paid attention to it until one of the ladies from shop told me.

Cultural difference is found in a lot of small things you do in your daily life, it's just like that scene from Inglourious Bastards when he is showing the German 3 and the British 3.

<comment><USER13> Yeah, there are lots of little things out there. My wife and I were just discussing how as Canadians, we don't perceive ourselves to have any accent and sound just like "normal" people on American TV, however after she speaks with US based co-workers or folks she meets when she travels, they instantly know after only a simple Hello or maybe 3 words that she is Canadian (and they point it out!). There are just certain words that we pronounce just a tiny bit different that we don't notice because we can say it the Canadian or American way interchangeably, but to Americans who aren't inundated with Canadian TV like we are with American media, we believe they can instantly detect any deviation from their culture's norms. It's kind of maddening because we can't detect the other way around nearly as easy.

She said she'd been caught on words such as Hello, Syrup and Being, to name a few. Perhaps it was just that she said Hello in a happier tone or with different emphasis? I'm not sure, that one seemed ridiculous.

Imagine trying to change all those words you don't even realize you are saying different unless you repeat it with the person multiple times.

<comment><USER10> >She said she'd been caught on words such as Hello, Syrup and Being, to name a few. Perhaps it was just that she said Hello in a happier tone or with different emphasis? I'm not sure, that one seemed ridiculous.

I lived in Canada for a bit, and I would have guessed sooorry, tomooorow, oout, aboout. At least that's what Ontarians sound like to me. They really stretch those o sounds.

<comment><USER13> I've lived in Ontario my whole life, North Western and South Western and I've never heard anyone sound like that. It seems like that's such a stereotypical Canadian accent, but I've never heard it and assumed it only existed in the maritimes, although on my recent trip to Halifax I did not hear it. Maybe it's a bit longer and I don't notice, but at least not like the overly portrayed version on TV.

<comment><USER10> It's very distinct from what you hear on TV. TV doesn't even do the right sound, nevermind the length, but it's definitely noticeable for an American.

It has some similarity to some oh the northern mid-west accents.

<comment><USER13> It's just so interesting to hear that I have an immediately identifiable accent when I don't even think I have one, so weird. It feels like you have a competitive advantage that I cannot easily copy or get back at you with :)

Perhaps this is also how folks with "real" accents think too (that they can't hear it), but I gotta feel like they can recognize that they sound different from the "people on TV". My wife did however tell me that in a phone conversation with a US colleague that she immediately heard herself sound obviously Canadian after saying something, but I think it was more phrase based, a question followed by "eh?". "Ok, I definitely heard it that time" she immediately said.

<comment><USER10> I think Australians and so on they know. The Canadian accent, compared to standard American, is much more subtle, but even after living in Canada for 2 years I got so used to it that I wasn't thinking about it in daily conversation. However when I left and came back a year later on vacation it was back to being obvious.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER14> Watching this video makes Bellingcat Skripal poisoining investigations even more hillarious.

</comment><comment><USER15> _Aaaargo!_

<comment><USER16> Could you please stop posting unsubstantive comments to Hacker News?

<comment><USER17> why does hn censor humor? that censorship is creating a feedback loop where creative individuals are left out of the discussions, which seem to be becoming more and more pedantic.

his post, while perfunctory, is still a reference to something culturally significant

imo flagging would be better reserved for more severe infractions and let the community decide (via vote) what is "substantive"

<comment><USER18> This isn't censorship (dang just made a request). Creative people and everyone is welcome; you just have to follow the rules. If you're really after puns and low effort posts then check out reddit, but even there some communities (like legaladvice or science) will "censor" a user for such posts.

<comment><USER19> Which specific HN rule bans humor?

</comment></comment></comment></comment></comment><comment><USER20> That guy at the end who changes into a vest is lols</comment>


Ask HN: iPhone? Blackberry? Android? Prayer? - How do you remotely monitor your startup? - <AUTHOR>

In the next week or two I plan to update my cell phone.
Besides the usual requirements, I want to use my new phone to monitor and tweak servers remotely via web and ssh.
I'm leaning toward an iPhone because I'm a huge fan of Apple, but I previously owned a Blackberry and was happy.
Anyone else using their phone to be a one-man IT army? Any recommendations? What is the ideal setup? <comment><USER2> I find that it takes a combination of tools, the key is a Blackberry and Nagios-

Nagios is a tool to automatically check machines for problems- It's very convenient in that I can highly vary the configuration for each machine- Dev machines might only be checked every 15 minutes, where productions machines might be checked every minute.

I have it parsing a variety of log files, and sending me emails when it finds different errors, or warnings.

The Blackberry is irreplaceable in this context- While I have an iPhone for my home calls and for testing mobile versions of our product, the iPhone mail client is next to useless.

It doesn't have any filtering or searching at all- That's fine for emails to friends, but it's worthless for monitoring.

The Blackberry, on the other hand, allows me to set a filter for certain messages, such as those with "ERROR" in the subject, to be "Level 1" messages, which I can treat differently.

For example, a normal message might vibrate the Blackberry, where as a Level 1 message is set to ring my BB increasingly louder until I manually check and clear it..

This ensures that if I'm sleeping, and we have a site-down-event, it'll wake me up and make me respond, even if I'm rather out of it.

With the iPhone, on the other hand, it's all or nothing.

Ideally, I'd prefer an app using the iPhone's push notification (whenever it arrives), that would allow me to use XML push requests for different alerts.. I could send an XML Warning, Error, or Fatal Error, for instance.

I'd settle for an email client with proper filters on either device, though..

The second point is search- The iPhone doesn't allow me to search through prior emails, which is ironic given the publicity of spotlight on the desktop. With my BB, I can search by day, sender, subject, etc; This is a necessity when trying to understand the context on a discussion.

Third- The BB email is much, much faster. Because it has real push email, I can get messages before they appear on my desktop- This is crucial for alerts. There is no good way to do this on the iPhone, without forwarding all my mail through Yahoo. Zimbra on the server side, connecting to a BES server is a cheap and reliable solution.

So far, I haven't been impressed with ANY terminal applications- The iPhone's screen is too small, and too much space is taken up by the virtual keyboard on these, and the BB doesn't include any useful keys, like pipe or backtick on the default screen. Normally, I use the phones to receive information, and then fire up my laptop + Aircard to fix it.

<comment><AUTHOR> Colin - Excellent, excellent post. This is exactly the type of feedback I was looking for. Definitely pushes me in the direction of a Blackberry. Thanks!

</comment></comment><comment><USER3> I prefer an n810 (great screen, hardware keyboard, Linux) with internet access over bluetooth and a small cell phone.

<comment><AUTHOR> Hmm... interesting. If I were to go that route I would probably get an Asus eee instead.

<comment><USER3> At least for me, it's important that the n810 fits in the pocket like an iPhone, unlike a netbook.

</comment></comment></comment><comment><USER4> iPhone

</comment><comment><USER5> iPHone. :)

<comment><AUTHOR> That's my top choice so far, but it's for emotional reasons more than anything. :)

Can you elaborate on your reasons for suggesting the iPhone?</comment></comment>


Real World Data: Wealthfront Tax-Loss Harvesting - <AUTHOR> https://blog.wealthfront.com/real-world-data-wealthfront-tax-loss-harvesting/?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+TheUpfrontBlog+%28The+Upfront+Blog%29 <comment><USER2> Is there a particular reason most US investing startups require an SSN? A lot of foreign ones seem to only need the address/id (it's usually okay if the investor is foreign).

If you're in Australia and want to try investing in a similar way check out [http://getfirststep.com](http://getfirststep.com).</comment>


Show HN: Imagine a search engine that removed top million sites from its index - <AUTHOR> http://millionshort.com/about.html <comment><USER2> The results are surprisingly good. I did some searches for recipes, and frankly without the top 1000 you really start getting some fresh hits. Entries by real people, rather than sites raking up recipes for a hit.

<comment><USER3> Wow, I didn't know until today that Google doesn't serve more than the top 1000 hits. (A result of me thinking, "can't I just ask Google to show me the million-first through the million-tenth hits?)

</comment></comment><comment><USER4> It's interesting to see popularity used as an inverse corollary with quality. Imagine a TV that skipped the most popular programming (goodbye American Idol), or a radio station that only plays non-hits.

Of course, there are great websites out there that are very popular (Wikipedia, NYTimes/WSJ, StackOverflow). I'd love to see a search engine with a better signal for quality than non-popularity (this search engine), or SEO (Google), but it's a fun start. :)

<comment><USER5> You have to be careful with that idea since it can still lead to poor quality. My campus's radio station (WREK Atlanta) seems to play just weird music for the sake of being weird. They never play popular music, but they rarely play any good music either.

<comment><USER6> WREK is just about the best radio there is. It's not weird for the sake of being weird. It's intentionally eclectic and far-reaching, sure, but the point is to broaden horizons. As much for the DJs themselves to broaden their horizons as the audience.

Disclaimer- former WREK DJ. :-)

</comment></comment></comment><comment><USER7> It means that our ranking algorithms have good recall but very poor precision. We value web page connectivity more than its content. We don't know how to teach machines to evaluate web page for its merit so we hope that a large number of Twitting, Liking and Plusing non-experts will approximate single expert. _Millionshort_ shows that this model isn't good enough.

<comment><USER8> I was able to rank for some moderately competitive terms just by paying someone minimum wage to go leave nice comments on dofollow blogs. The whole "Content is king" meme is a joke, really.

<comment><USER9> Interesting. How do you find blogs that don't implement nofollow?

<comment><USER8> Plenty of blogs promote themselves as DoFollow Blogs. It's a bit sad, really. They basically trade their "link juice" for your insincere, contrived comments.

<comment><USER10> Which in turn can help them rank for more terms.

</comment></comment></comment></comment></comment><comment><USER11> This is the first off-brand search engine I've seen that's, in some sense, cooler than Google.

For one thing, the huge miasma of spam websites that dominates the SERPs just isn't there -- I hope this lights a fire under Google's butt and people see another world is possible.

<comment><USER12> It reminds me of why I first moved to Google from Yahoo/Webcrawler/Altavista/etc in the first place.

<comment><USER13> Yup. Since this is trivial for Google to copy, it's unlikely that it'll actually disrupt Google in any way, but still...very cool.

<comment><USER14> The technology is trivial, but the reasons behind it and philosophy that drives it aren't.

Google isn't going to come along and say, "oh, guys, you're right...Wave, Buzz, Google+ all a mistake. Social Graph..Mistake!. We're sorry. We are rolling everything back to 2004"

Google has been increasingly more frustrating for power users over the last couple of years. People are looking for alternatives. DDG is one, this is another. Power users matter, and not just because they spend more time inside an application, but because they recommend it to others.

A perfect example of this is firefox. I used firefox a lot, and then switched to Chrome. I've since switched my mother, grandpa, brother, girlfriend, sister and too many friends to count to chrome. I now use DDG for all my searching needs via the bang syntax (!, !g, !w, etc). It's only a matter of time before the rest of my family follows along.

<comment><USER15> Ah, I was unaware of the !g bang syntax, that's a great one. I'm also fond of !archwiki, and all the language bangs. DuckDuckGo is the only search engine that I know of that could take over a significant portion of search traffic. By significant I don't mean tens of millions, but I mean a dedicated userbase who use it in leue of Google.

<comment><USER16> Just out of curiosity, what does DuckDuckGo do that conventional keyword searches in Firefox or Chrome don't do?

<comment><USER14> I have DDG set as my default search engine, but can you do the following:

! fuji Ames => takes me to Fuji Steak House in Ames Iowa directly

!w Sushi => Takes me Directly to the article on Sushi on Wikipedia

!m 801 grand ave Des Moines => takes me directly to google maps at the address.

I still probably do about 30% of my searches using !g kicking me straight to google for search results.

<comment><USER16> But how is that different from regular keyword searches? If I want to pull up the Wikipedia article on sushi in Firefox, I enter "w sushi" in the URL bar. If I want to read the Arch Linux wiki article on pacman, I type "arch pacman". If I want to watch Friday by Rebecca Black on Youtube, I type "y Friday", etc. I feel like there's more than a 50% chance that I'm missing something. What does DDG do that browsers don't?

<comment><USER14> I typed in "w sushi" into chrome and I got [https://www.google.com/search?aq=f&ix=ucb&sourceid=c...](https://www.google.com/search?aq=f&ix=ucb&sourceid=chrome&ie=UTF-8&q=w+sushi) which google set to my default. I expected to get "<http://en.wikipedia.org/wiki/Sushi>

Do I have to setup all these different keywords to work? If so that's the difference. It's essentially a pre-configured command line with literally hundreds (thousands?) of predefined syntax for searches.

<comment><USER16> Yes, I set those keyword searches myself. So the difference seems to be that with DDG, you just have to learn which shortcuts exist rather than customizing them yourself. That doesn't really sound like it saves much time, but it sounds like the real advantage of DDG is that the act of reading through the DDG list of keywords would probably give me ideas of useful time-saving shortcuts that I'd never think of making on my own.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER17> I'd be fascinated to see the kind of SEO that would go on if this took off.

"Bad news-- we're a top 100 hit for several of our main keywords. We'll have to change our URL scheme again."

<comment><USER18> We can call it Search Engine Pessimization!

<comment><USER19> Bravo!

</comment></comment></comment><comment><USER20> This reminds of searching the internet in the 90's, I'm finding results from pages I haven't visited or heard of before now.

This is really refreshing.

</comment><comment><USER21> This is a breath of fresh air - I'm loving the unpredictability of the top results! It's like flicking through a new set of 1000 tv channels in a different country.

<comment><AUTHOR> The positive feedback is awesome.. Thx

</comment></comment><comment><USER22> Would you share some implementation details.

What's your source for the top million sites; where do you get your site list from for the other results?

</comment><comment><USER23> Who needs to imagine it? It's here.

Cat is out of the bag.

What is the Alexa list good for? Answer: Filtering out the boring, money- grubbing commercial sites. A truly GREAT idea.

A return to the good 'ole days. The non-commercial web.

Many young people who love today's www never got to experience it as it was before it became overrun with Google-ization and auto-generated garbage.

Take the ball and run with it. We ca reclaim the web. This is only the beginning.

</comment><comment><USER24> Wow... It felt like using Google 10 years ago. I think you are onto something.

<comment><AUTHOR> Wow.. awesome comliment..

<comment><USER25> Usually don't search unless I'm looking for something in particular, but just played with this for a good 15 minutes running random queries. The results are really good and at the same time I'm discovering sites I'd never otherwise see with Google.

</comment></comment></comment><comment><USER26> Turns out removing the top million results from a search for Google... still returns google. Or google.com.au to be precise.

It's a cool idea, but I'm not sure it's working. I tried "american history" but it wouldn't return anything at all if I changed the "Remove the Top" dropdown.

<comment><AUTHOR> Good feedback. Right now the filtering is not 100% working for non .com, .net, .org, .co.uk domains. Still working on it.

</comment></comment><comment><USER27> There's one pretty big flaw with this approach... For certain searches that do not have "millions of results", you get completely unrelated results.

If I search my name then the results are for names similar to mine, but not actually my name. This makes it completely useless for searching my name. I would think that there are many searches with this problem.

I think there needs to be some kind of weighting system used that dynamically decides the cutoff point. One million is a huge over-generalization for all search terms.

<comment><AUTHOR> The number of results aren't a factor. We are removing results from sites that are in the list of top 1 million most popular web sites. Hope that helps.

<comment><USER27> Ah, this seems to be the case. It looks like you are dropping my name down to "thousand from results" despite it still saying "million from results".

Very nice!

</comment></comment></comment><comment><USER8> I like this idea a lot. I came across a nice, concise explanation of a Buffer overflow

[http://www.apolis.org/index.php?option=com_content&view=...](http://www.apolis.org/index.php?option=com_content&view=article&id=81:what- is-a-buffer-overflow&catid=44:faq&Itemid=62)

<comment><USER28> This is a copy of the Wikipedia article on buffer overflows.

<comment><USER19> Looks like this idea isn't totally in the free-and-clear from content farms.

<comment><USER10> Yeah except now you get the content farms that don't know how to rank. A competitive spammy term like "pay day loans" still shows plenty of low quality sites because of the crazy number of sites looking to cash in on the term.

</comment></comment></comment></comment><comment><USER29> It's a similar measure that's often used in NLP. Sentences, documents etc. are usually stripped of common or popular terms first and the remaining ones tend to have higher information value.

It's not entirely a surprise that it works for meta-language constructs like the web and site popularity.

<comment><USER30> Uhh,

1) I am basically certain that it doesn't work. Imagine something this simple actually did work in general. Do you think Google, Bing, etc, wouldn't have implemented it?

2) I think the analogy is extremely flawed. nytimes.com is by no means a web page equivalent to "the" or "a" in language. Articles, pronouns, etc, don't really carry meaning. Despite being popular, nytimes certainly does.

<comment><USER29> I don't know. I haven't really spent a lot of time looking at this to get a good feel of the quality. But consider this, NYT tends to be a secondary source: e.g. book reviews (books are the primary source), science news (scientific papers are primary), business news (market movements and press releases are primary), etc.

Now consider, if I'm interested in some particular thing in science, is it better to get the NYT science reporting or just get the professor's publication and research page on their university website? Filtering out the top-n sites is more likely to turn up the professor's site near the top of the pile rather than after all of the popular sites' second and third level reporting.

Is this better? Depends on the audience. A "popular science" goal would argue the former is the better as science news simplifies, abstracts and popularizes complex science (with varying degrees of quality) while a scientist would prefer the latter.

</comment></comment></comment><comment><USER31> The results feel actually fresh. It's removing the consumerism layer of bullshit that google serves us everyday. Also wondering if that has something to do with the "bubble" that google creates around us based on our search history and social network information.

Thanks for that.

</comment><comment><USER32> Searching for 'python global interpreter lock' yields some interest blog articles describing the problems, also some related articles about approaches to the C10k problem with python (preforking, worker processes, etc.)

A++ would search again.

</comment><comment><USER33> I'm really liking this. Instead of being bombarded with content that's just blasted with keywords, I get relevant well-written articles. Not only that, but no more W3Schools in my SERP's. The chance to read an article that's written with humans in mind, instead of Google is more than enough reason to spend some more time using this.

</comment><comment><USER34> You think that top results in Google and other commercial search engines are always ranked based on "popularity"?

It would be harsh to call this naive, but it shows a serious lack of SEM and SEO knowledge. Ever heard of "paid placement"?

Many years ago when Digital's AltaVista was our main search engine, it was becoming loaded down with paid placement.

The results were polluted.

Google eventually became the "clean" solution.

But now it's Google that is loaded down with all sorts of commercial crud, much of pointing to Google acquisitions.

And paid placement, among numerous other strategies, new and old, still exists.

The simplicity of millionshort is brilliant.

Filter out the crap.

</comment><comment><USER35> Add a way for me to put this as my search engine in my firefox search bar.

Please.

EDIT: In trying to accomplish this task I found an add-on that lets you do this for anything.

([https://addons.mozilla.org/en-US/firefox/addon/add-to- search...](https://addons.mozilla.org/en-US/firefox/addon/add-to-search-bar/))

<comment><AUTHOR> Awesome.

</comment></comment><comment><USER34> Here's what Dropbox thinks about power users:

WSJ: What's next?

Mr. Ferdowsi: We continue to focus on actually solving problems that real people have and not being distracted by what power users want.

Google has made clear what they think about power users:

No + operators in search.

No web-based code search.

No Google Labs for the public.

etc.

Plenty of wood behind the Google arrows, but all the cool ones have been cast out of the quiver.

Just what kind of targets is Google aiming at nowadays?

Millionshort I give you +999,999.

I would give you +1M if you took out the AdSense and PlusOne javascript.

This has been a long time coming.

Alas, DDG and other alternatives are all about _money_.

Search is about _discovery_.

</comment><comment><USER36> Well that's one way to break out of the filter-bubble/echo-chamber I suppose. If only our best search technology was based on something better than a popularity contest :(

<comment><AUTHOR> Popularity I think is important. But not at the expense of relevance. It's not a easy nut to crack.

<comment><USER37> Why? If I search, say, for a game review, I don't care whether it comes from a popular website or a blog no one reads. In fact, the topmost websites are more likely to be biased, since they try to appease everyone and they also have strong relationships with publishers. The blog no one reads is nearly guaranteed to be honest (if not well-written).

This holds true for most topics I can think of. Moreover, if I ever need to read Wikipedia and such, I _already know_ about those websites, and I can go there directly - no need to search. Shouldn't web search engines act like discovery tools?

<comment><USER36> > Shouldn't web search engines act like discovery tools?

If your business model is based on advertising that depends on masses of page views to generate value, then no. You want to be as generally useful as possible so that e.g. people use you as an (extremely inefficient) DNS service.

(Google Labs has a single optional feature available for search. Perhaps their arch doesn't make 20% or Labs projects a good fit for plugging in extra fancy search features?)

</comment></comment></comment></comment><comment><USER38> And of course, W3Schools still manages to show up, thanks to their multiple crazy subdomains: <http://cl.ly/GFup>

<comment><AUTHOR> Your're right. We strip www from the domains. We need a better function to rip out the domain from a URL. Just haven't had time to cook one up yet.

<comment><USER39> Try out <http://publicsuffix.org/> \- that plus a custom suffix list of overrides works wonders!

</comment></comment></comment><comment><USER40> After a few test searches, this is surprisingly effective for things which I had resigned "un-findable" because of poor Google results. This is most apparent on non-technical things, in this case specific Jazz chord fingerings for a guitar class I am taking.

I am very interested as to what comes of this, or rather what is influenced by its implications.

<comment><USER41> Guitar chords / tabs / lessons are a terrible SEO spam offender... Look at that! I finally found an accurate transcription of "Bohemian Rhapsody"

</comment></comment><comment><USER13> Man, I love this thing. I've already found a bunch of interesting links on path tracing. Bookmark'd.

<comment><AUTHOR> Sweet. Glad to have helped.

</comment></comment><comment><USER42> What is the ranking used for the top million sites? A search result for "Australia" returns as the top result <http://australia.gov.au>, which Alexa ranks as 20,615 globally. Actually, a lot of the queries I tried returned Australian sites.

[http://millionshort.com/search.php?q=australia&remove=10...](http://millionshort.com/search.php?q=australia&remove=1000k)

[http://millionshort.com/search.php?q=somalia&remove=1000...](http://millionshort.com/search.php?q=somalia&remove=1000k) \-- another Australian site.

<comment><AUTHOR> Right now the filtering is not 100% working for non .com, .net, .org, .co.uk domains. So, .gov.au isn't being properly filtered yet. Soon.

<comment><USER43> The Public Suffix list <http://publicsuffix.org/> maintains a list of domains you need to filter on.

</comment></comment></comment><comment><USER44> I'd really like to see randomization instead. Return results picked randomly from within the top 10 million or something.

<comment><USER45> I'm not sure this is a great idea. Predictability is a staple of a good user experience. Getting different results for the same query between users or sessions is bound to lead to broken expectations and frustration.

<comment><AUTHOR> Think of this as more of a discovery engine. And predictability takes the fun out of discovery.

<comment><USER46> Great point. You might have a hard time competing with google on relevant results but you might be able to beat them on discovery, like a stumbleupon search engine.

</comment></comment></comment></comment><comment><USER47> It's amusing to see all the SEO "experts" that don't make it into the top million:

[http://millionshort.com/search.php?q=seo&remove=1000k](http://millionshort.com/search.php?q=seo&remove=1000k)

<comment><USER48> It removes the top million in general, not per query.

</comment></comment><comment><USER49> A real serendipity engine. Absolutely great, thank you. I'm finding tons of interesting products and ideas by searching the most banal things :)

</comment><comment><USER50> I would prefer if my previous search was populated in the search box after completing a search (since I might want to try the search with a different filter).

It appears you have an "off by one issue" in the sidebar. There's always a blank entry in the list of ignored sites.

Filtering does not seem to be working (or I don't understand it). Searching on "chicken" produced the same results with 1million or 100k removed.

<comment><AUTHOR> Will add that feature. Thx

<comment><USER50> I neglected to mention... Awesome idea!

</comment></comment></comment><comment><USER51> Thankyou! I can see this being something I regularly use.

It may be a simple idea, but its something nobody else has done before, and I think the creators deserve a lot of credit for coming up with and implementing it. I hope they manage to get something from it. I can see that if the site becomes popular it will just get copied by other search sites.

</comment><comment><USER52> This is pretty amazing. I didn't know that the old internet was still there! This may become my new favorite search engine.

</comment><comment><USER34> "Quality" is subjective.

More relevant is _accuracy_, i.e., you get what you specify via search operators, and results are not influenced by all of Google's silly "factors". You know what you're looking for and how to frame the query. But Google assumes you're dumb and thinks it should decide for you.

Alexa Top 1M is a nice filter because the data comes from the Alexa Toolbar which only the most braindead web users would have installed. So you are in effect avoiding sites that the web's most braindead users would often visit.

Ranking sites based on "popularity" is great until you reach the point where the majority of users are not very intelligent. (cf. search engine users in 2004 with search engine users today.) When you reach that point, you get results where "quality" is determined by idiots (and SEO hats), not a group of intelligent peers.

</comment><comment><USER53> It's like a hipster search engine. It's only interested in things before those things are cool.

</comment><comment><USER54> Is it safe to assume that this is how Google's search results would look if nobody did SEO?

<comment><USER55> No, not at all. Google is driven by many signals that SEOs try to optimize on, but the methodology is still the same. Top-ranking sites have high-quality sites linking to them, good content, and are supposed to not look spammy (though that's debatable).

I don't know how this engine ranks but I assume it's a similar system, they're just chopping a good portion of the results out. For what you said to be true, the top 10K/100K/1M rankings would have to be there entirely due to intervention from an SEO, and that's just not the case. The Wikipedia's of the world have enough going for them that they don't need SEO, so they'll always show up in Google, and never in this engine.

</comment></comment><comment><USER56> Funny, on my first query I found an obscure HN scraper:

      http://tazod.com/

<comment><USER57> Wow I think this is the exact idea I had for "HN Time Machine" - basically a thing to extend the life of the newest page and speed up the movement of stuff through the home page (ie. I think the HN new page moves too quickly and the home page not quickly enough)

</comment></comment><comment><USER58> Exactly what I was thinking in 2001 => <http://www.halfbakery.com/idea/The_20Other_20search_20engine>

Glad to see someone did it now...

</comment><comment><USER59> This is a great idea and I see myself coming back to this. It's a shame that a little blog on tumblr or blogspot gets taken out because it's under a big name domain - but this has spam related benefits too.

Great work!

<comment><AUTHOR> Maybe we can set up a Webmaster tools sort of submission process for inclusion.

</comment></comment><comment><USER60> Wow some of the content there is great. Forget about searching the deep web. For me deep is the real gems buried under the first 100 or so results where stuff actually gets interesting!

</comment><comment><USER61> Only a little thing, could do with maintaining query strings between pages. It lost my query string and returned no results when I changed the drop down without me noticing.

<comment><AUTHOR> Good catch. Will fix. Thx

</comment></comment><comment><USER62> I have been wanting something like this for a while. It's even on my todo list. Thanks for saving me the work. I'll be using it all the time!

</comment><comment><USER63> How did you build this? Are you indexing the entire web yourself? Or are you using Google's index/removing the top 1 million based on domain?

<comment><USER64> I think search APIs like Yahoo BOSS allow you pass arguments that contain a black list of domains. I think it's the 'sites' argument that may be used like this: &sites=-google.com

<comment><AUTHOR> You are right, but they won't allow the list to be 1 million sites long. You are talking 15 megs of data per data in plain text per request.

But I like the idea of being able for users to, via a setting perhaps, add their own list of deny/include sites.

Thx for the comment.

<comment><USER65> Isn't this under Blekko's domain of ideas with the slashtags letting you include particular sites?

</comment></comment></comment></comment><comment><USER66> I just tried it with a search for some competitive intelligence. I used the 100K removal option. I found a competitor in another country that had not made the top 2 pages on Google. It confirms that others are launching something similar to what I am building... but also the fact that it doesn't bubble to the top on Google means that the market space is not dominated yet.

</comment><comment><USER67> I love this! And am totally going to use it. Removing the top "thousand sites" removes pretty much all the sites I WISH I could have filtered from my Google results anyhow (ehow, w3schools, etc).

One request: please keep the search text in the form field after clicking "search". Just so users can search the same thing multiple times with different values in the "Remove the top" drop-down.

</comment><comment><USER68> Thank you - very refreshing! DuckDuckGo should implement something like this just for the spirit of it.

The web just got more interesting. :)

</comment><comment><USER69> Doesn't make a dent in the travel site spam, unfortunately. though I might use this just to permanently remove About.com...

<comment><AUTHOR> A few of the suggestions included the ability to set include/exclude sites which I think we'll add.

</comment></comment><comment><USER70> Just re-found a site I was looking for but couldn't find with google the other day. This could definitely be helpful.

</comment><comment><USER71> Not sure if I found an anomaly or what, but a simple search of "Privacy" returns results from thesaurus.com, merriam-webster.com, truste.com, kelloggcompany.com, and many more that are all in the top few thousand according to QuantCast and Compete.

Great idea though, will definitely try this out some more.

<comment><AUTHOR> merriam-webster.com redirects to m-w.com - looking into the others.. Sometimes if a site has header redirects it gets lost in the filters. Thx for pointing this out.

<comment><USER71> ahh that explains it. I'm actually working on a personal project right now and this could help out quite a bit, so I am excited to see where this goes. Best of luck!

</comment></comment></comment><comment><USER72> Is it just me, or are the results fairly congruent with standard results from a search engine?

<comment><AUTHOR> They are to a degree. We just remove remove the top million most popular sites. So, searching for "social network" would normally yield Facebook.com, but with MillionShort you discover a social network that that didn't make it past the noise.

</comment></comment><comment><USER73> Great idea, although I think if you could explain it a bit better you could avoid the confusion like several of these comments are showing. I like how my Hacker Newsletter project shows up #2 when searching for Hacker News. :)

<comment><USER74> And how do you determine the "top" million sites?

<comment><USER75> Check out:

<http://www.alexa.com/topsites>

</comment></comment></comment><comment><USER76> I think remove results with my search term in the domain name and this would be perfect!

For example I searched for how to start a garden and I can guarantee that startagarden.com is junk. But indie see some useful advice from small blogs etc

<comment><AUTHOR> How would you like to see this work?

<comment><USER76> Well for example I searched for "grow a vegetable garden" and I still see a lot of results from URL's like:

<http://howtogrowavegetablegarden.net/> <http://www.grow-your-own-vegetable- garden.com/>

And I just know any domain name optimized for a certain search term is going to be garbage.

I guess a simple version of this feature is you'd have a setting: "Exclude domains that contain my search term".

When the user clicks that you'd compare the domain name (removing all special characters) with my search term (also removing all special characters and white space). Maybe compare via edit distance and exclude if it passes a threshold?

Although edit distance might not work too well, perhaps looking at the longest common substring and if it's > say 90% of the length of my query exclude it?

I guess it would take some playing around. But there should be a good algorithm to exclude domain names very similar to my query.

</comment></comment></comment><comment><USER77> I'm getting odd results with the following query :

Search String : Ruby

Remove From Top : 1000 & 10000

In both instances, the top hit is <http://www.ruby-lang.org>, which is also the top hit from both Google and DDG.

Am I missing something?

edit: formatting

<comment><USER78> I think that removes the top 1,000 or 10,000 across all searches, not just "ruby".

</comment></comment><comment><USER79> This is a cool search engine for discovery, but it defeats the purpose when you are looking for a location. Do a search for "facebook", you will not get any result that links you to facebook.com .

<comment><USER80> I've seen people do this a few times, and don't understand why they do it (Google 'facebook' or 'twitter' and then click the link to get to the site instead of typing 'facebook.com' or 'twitter.com' into the address bar). Can you explain why? I'm truly curious.

<comment><USER81> Couple reasons I've encountered..

1\. User doesn't understand the WWW. They believe Google is THE gateway to everything internet.

2\. User doesn't know (or has trouble typing) the exact address. Google tends to have the authoritative result first for popular sites like facebook, amazon, twitter..

<comment><USER82> I also often don't know the exact address, but browsers autocomplete URL for me. I think it's a fair bet those who search for Facebook already visited Facebook. Why search then?

<comment><USER81> I prefer to go straight to a site, rather than detour through a search engine. That said, I use chromium, and use the address bar like a search engine. Sometimes the auto-complete half-way through typing is the search rather than the site. If I don't notice..

</comment></comment></comment></comment></comment><comment><USER83> I would like to see the search engine adhere more strictly to quoted search terms. It seems that they are partially ignored, which gives it some of the same problems that the major engines have.

</comment><comment><USER84> Please add a favicon so I can see in my (icon only) Bookmarks Toolbar.

<comment><AUTHOR> favicon coming right up. Minutes away.

</comment></comment><comment><USER85> Good idea. It's about time that search engines route around the power-law distributions of popular sites, popular bloggers and personalities to find the gems otherwise buried in the noise.

</comment><comment><USER86> > Imagine a search engine that simply removed the top 1 million most popular > web sites from its index. What would you discover?

A lot of my competitors who are still on the first page of Google results.

</comment><comment><USER87> Wow that is pretty awesome. I reached some results I want that I could not find via popular search engines with hours of searching. Believe it or not, this engine is changing my life.

<comment><AUTHOR> Glad that code is being put to good.

</comment></comment><comment><USER88> Very interesting. I was pleased with the results and have already added this site to my Chrome bookmark bar, right between my Google search and Hacker News icons.

</comment><comment><USER89> Well, in a rather meta turn of events, searching for my username on this returned a link to hackerbra.in, which appears to be some kind of HN mirror.

</comment><comment><USER90> If you search for "google" and remove the top million results, you still get google main page (in this case, the one for australia and india...)

</comment><comment><USER91> The site's way too wide on my netbook, 1024x600. Also, the list of domains removed from the results covers up part of the results themselves.

</comment><comment><USER92> Another Cool feature could be to exclude sites that use Adwords or Paid Search from the list too. Then it would really just be legit sites.

</comment><comment><USER93> It doesn't appear to work. I did a search for aspirin and the top match returned by this is #5 doing the same search with Google.

<comment><USER94> Top million sites, not top million results.

<comment><AUTHOR> Correct. Top million sites.

</comment></comment></comment><comment><USER95> If this becomes popular, at some point results would disappear since unpopular sites will be pushed into the first million.

<comment><AUTHOR> I guess the thought would that graduated to some level of critical mass. Sort of like a kickstart program for sites.

</comment></comment><comment><AUTHOR> 8 hours later, we just launched our first re-design. Thanks for all the great feedback and support. More to come.

</comment><comment><USER96> I really like this. Right away I found some new sites about that I hadn't seen before with interesting content.

</comment><comment><USER97> As someone learning web development, I'd love to get some insights into how one could build this.

<comment><USER98> Not knowing anything about what they do, the hack-ish way you could do it is to use Google CSE (custom search engines) to add a list of negative domains. Where to get the list of top 1 million domains? Probably from Quantcast here: <http://www.quantcast.com/top-sites-1>

</comment></comment><comment><USER99> Thanks.

Google has really killed the discoverability of the internet for me. I will be experimenting with this.

Best of luck.

</comment><comment><USER100> This is an incredible breath of fresh air, what an odd thing to say.

</comment><comment><USER101> Removing Wikipedia might be a mistake. Otherwise, it's great.

<comment><USER37> Why would anyone want Wikipedia in search results is beyond me. If I want to read a Wikipedia article, I can just search Wikipedia itself. I know the kind of information it has, so there is no point in "ranking" it against other websites.

<comment><USER102> I'm not sure if this is still the case, but in the past Wikipedia's search engine was terrible and it was actually easier to google "X wikipedia" or "X wiki".

</comment></comment></comment><comment><USER103> i searched for my site and in teh goog i get first page....here i found nothing....so for me this = no good. I understand the base but i dont understand the result

</comment><comment><USER104> Should I be depressed that I'm the top hit for my own name?

</comment><comment><USER105> I wonder if there's an analogous hack for social news?

</comment><comment><USER106> Now if I could add it to firefox's search bar...

<comment><USER107> I came here to make the same request.

And I am loving this.

</comment></comment><comment><USER108> very interesting hack. thanks for doing it

</comment><comment><USER109> non-popular websites will start seeing some good traffic suddenly. It would be confusing for them :)

</comment><comment><USER110> my website ranked 1st? guess i need to work harder T_T

</comment><comment><USER111> Searched my name... Got my website

:(

</comment><comment><USER112> you should some how incorporate hipster into the search site's name.

</comment><comment><USER113> I searched for "Hero Academy" and the first result was Google's 5th result, a site called Hero Academy with the url "hero-academy.com". That's not very "million short", IMHO.

<comment><USER114> The site removes the top million sites by popularity, not the first million results returned by a Google search. "hero-academy.com" is likely not in the top million most popular sites.</comment></comment>


Apple Tablet is all but confirmed - <AUTHOR> http://www.lockergnome.com/griffin/2009/12/23/apple-tablet-is-all-but-confirmed-for-january-26-2010/

<comment><USER2> Meaning it's still a rumor.</comment>


Outsourcing Marketing: How $300/mo on ODesk Boosts Our Traction by 14% - <AUTHOR> http://www.startupmoon.com/outsourcing-marketing-how-300mo-on-odesk-boosts-our-traction-by-14/

<comment><USER2> I am confused. As marketers, surely you are familiar with the old adage "you get what you pay for." I feel this applies directly in outsourcing -- skilled labour does not come cheap.

How can you be confident in results for such large data research pulls (which take time) when effectively scraping the bottom of the barrel of freelancers with such insultingly low outsourcing budgets?

Furthermore, I don't particularly agree with the assertion that you can fully decouple the research process from domain expertise. Should a problem arise, understanding of the project, background, and goals is key to deciding a course of action, especially if you're outsourcing some analytics.</comment>


How can I tell if a plant given to me is patented? - <AUTHOR> http://gardening.stackexchange.com/questions/3808/how-can-i-tell-if-a-plant-given-to-me-is-patented <comment><USER2> For what it's worth:

When people think about plant patents, they're usually thinking of the Monsanto suits. I don't know that there aren't crazy patent trolls out there suing gardeners, but as far as Monsanto is concerned, I don't think gardeners have much to worry about.

If you read the Monsanto suits that have been published, the behavior Monsanto pursues is planting unlicensed Roundup-Ready seeds and then spraying them with glyphosphate-based ("Roundup") herbicides. You can spray Roundup without a patent license. You can probably plant RR crops without a license. But if you do both (commercially, at least), Monsanto sues.

The point of RR seeds is that they resist Roundup, which is a broad-spectrum herbicide that will kill non-RR crops. In the commercial suits, it becomes tricky to argue that you planted RR crops unwittingly when you are later shown to have sprayed them with an herbicide that would have certainly ruined your harvest but for inbred RR resistance.

The patent they have on the whole RR system may be totally invalid; I'm hoping to have the presence of mind not to end up litigating that point. I'm just saying, the likelihood of accidentally become a lawsuit target appears like it might be overblown in some ways.

<comment><USER3> I agree that people _usually_ think of Monsanto suits in conjunction with plant patents, but the point that the article raises is valid. How can one tell? If you can tell, don't we _all_ owe money to the heirs of whoever invented corn, and whoever invented wheat? After all, copying is theft...

<comment><USER2> Like I said, if you want to argue that there is no possible validity to a plant patent, that's something I'm not prepared to litigate. I'm just saying that accidentally planting a RR plant in your garden or even your commercial farm probably won't get you in legal trouble, because you aren't going to spray them with glyphosphate.

<comment><USER3> How on earth did you get the idea that I'm arguing against the validity of a legitimate, duely legislated and properly instituted legal entity like a plant patent? In point of fact, I argued the exact opposite. I feel quite offended that you attribute the exact opposite of what I wrote to me. I wrote that copying is theft, and I implied that we all owe some money to the heirs of the inventors of agriculture (there were probably at least 5 such inventors) because of the continued use of said inventor's property.

What's your beef with that? Are you a pirate, or one of these anti- Intellectual Property heretics/crazies?

<comment><USER2> I'm afraid you may have figured me out.

</comment></comment></comment></comment></comment><comment><USER4> I'm not a patent lawyer, so maybe I'm wrong and someone can correct me, but all this "how can you patent a seed" whining is bullshit.

The Monsanto patents are for the method of using a particular herbicide in combination with the modified plant seeds. If you innocently get some Monsanto genes into your organic soybean seeds, then you would almost certainly not be infringing the patent any more than I could be infringing DVD patents by using a stack of DVDs to prop up a wobbly table.

Has Monsanto ever sued anyone who was not trying to use the combination of the GMO crops _and_ the herbicide they are tolerant to? I have never heard of such a situation.

I've yet to meet anyone who claims to think it's bad for Monsanto to have a legal monopoly (patent) on this process who doesn't also think that no one should be using it, because it's either screwing with nature to use gene splicing to feed the world, or maybe just because not having our crops ruined by pests makes us soft and unworthy or some crap.

Read Mark Lynas's post explaining "I am also sorry that I helped to start the anti-GM movement back in the mid 1990s, and that I thereby assisted in demonising an important technological option which can be used to benefit the environment."

[http://www.marklynas.org/2013/01/lecture-to-oxford- farming-c...](http://www.marklynas.org/2013/01/lecture-to-oxford-farming- conference-3-january-2013/)

<comment><USER5> @tptacek's opinion is much better without resorting to phrases such as "whining is bullshit".

> _Has Monsanto ever sued anyone..._

As a car driver in a crowded city I bend the rules all the time. Just this week alone there were several rules I broke that could have lead to my license being suspended, but because the city is so crowded, the probability of being caught for breaking rules (such as going around a tram pedestrian refuge, or forcing the yellow light) is close to zero, unless you end up in an accident.

There is a difference however in what is _legal_ to do and what you can get away with. Our current legislation systems tends to make criminals out of all of us. The example I just gave is a bad one too, because I know exactly what rules I'm breaking and I know exactly the outcome if I get caught, whereas with infringing patents you never know exactly what patents you infringe, or the outcome of that.

There's also the issue of selective enforcement. The law should be the same for everybody, otherwise if everybody is a criminal by default, the law gets used to suppress people that are undesirable.

It doesn't really matter if Monsanto never sued anyone for using their patented crops without the usage of the Roundup herbicide. The bigger issue is that if you receive some seeds from a friend, planting them in your garden will leave you on the wrong side of the law and you might not even know it.

</comment></comment><comment><USER6> I agree that this is insane, but I'm not sure what the solution is.

With most patents, the company that developed the technology to be patented has a unique advantage even without the patent. They have a better technical understanding of the technology than the competitors, and they can incorporate it into their products faster.

With crop patents, the patent seems far more necessary as an incentive to innovate. If Monsanto can't patent their new seed, then anyone else who wants to can also sell the seed, and Monsanto will never recoup the R&D costs. Under this scenario, I don't see why anyone would develop new plants.

Assuming we want to encourage selective breeding/genetic engineering of new plant species, what incentives can we devise that don't have the problems posed by patents?

<comment><USER7> > If Monsanto can't patent their new seed, then anyone else who wants to can > also sell the seed, and Monsanto will never recoup the R&D costs. Under this > scenario, I don't see why anyone would develop new plants.

Could the same apply to GNU or Linux or any other free software project or is it far different thing?

<comment><USER8> To develop free software, all you need is a PC, Internet and free time.

To develop free GM plants, you need workers, material (host microorganisms, plant fertilizer), green houses, lab time, machines (PCR, sterile workbenches, ideally a sequencer, PCs, maybe a fun electroporator, etc.), space for growth etc. Also, the laws are strict - you can't just create GM-organisms in your backyard, depending on the country you're in you have to be in a certified lab with correct procedures for waste disposal etc.

The upfront costs involved in developing GM are much, much higher. But it's definitely not impossible - there's for example an open source PCR machine for self-builders, I can see more machines like that coming up...

</comment></comment></comment><comment><USER9> Is this question what it seems? Can gardeners really get into patent trouble from planting a seed?

<comment><USER10> Short answer: No.

Long answer: The question is sparked by a recent court case where a farmer - named Bowman - purposefully used unlicensed technology on a commercial scale. If you are doing it by accident, or aren't doing it on a commercial scale, you're perfectly safe. (Some reporting in the popular press makes it sound like Bowman was unaware of his infringement or that it was an accident - a lot of people are freaking out at the idea of Monsanto suing a farmer because some patented seeds happened to be found on his property. If you read the court case, it becomes clear that nothing of the kind happened, and that if it had the case would have been thrown out.)

Really, it's just like every other bit of patented tech. If you're operating a factory pumping out unlicensed Rolex watches, you've got some potential legal liability. If you bought a fake watch in Bangkok, you don't. While I personally have some very serious qualms about the patents on GM crops, the current legal situation is, honestly, pretty sane. Hobbyists are perfectly safe, as are farmers acting in good faith (ie, they plant seeds believing them to be unpatented, or their seeds are contaminated by patented seeds blown across their property line). Activists like to construct scary hypotheticals, but the truth is rather more prosaic.

<comment><USER11> I think your first paragraph would likely be relegated to spin. While your oversimplification of the problem makes it seem acutely cut and dry the short response is: it's not that simple.

Like all other constraints of life in general, I think going lax on things like a separation of an ill corporation and the direct corollary effect it poses to something nature has originally derived is treading in unknown territory. The fact that this simple question would have been laughable at best 15 years ago showcases the downward spiral into everything being a legal matter in the not so distant future. You may think this is counterproductive activism but skim the history books. Look around you and reflect on the choices that others have made to put your in your first world situation. That can quickly change if there aren't people who keep those ideals in check and balance out the corporate tendencies that often times spins out of control when left to its own device. That is a path where morals take a back seat to suits and dollars more often than not.

</comment></comment></comment><comment><USER12> Have the person providing the plant indemnify you against future litigation.

Someone knows whether the material is patented or not, but they are up the sales chain from you. If they agree to indemnify you against patent litigation, then they know it's not patented. If they won't, then they either didn't do their job in getting similar assurances from their supplier, or they are hiding something.

In general, you work backwards through the stream of commerce indemnification clauses.

It's ridiculous overhead for gifts between friends, of course. That's why, for other areas of the law, like theft, we realized hundreds of years ago that you should give people some automatic protections if they were good faith purchasers. (Not 100%, jurisdictions vary, civil and criminal are different.)

Not sure why we don't borrow similar provisions for patent law, but I guess Bowman v. Monsanto could do that (if it's not simply remanded on a technicality).

</comment><comment><USER13> The mere fact that this question was asked is a sign of things being badly broken in the patent area.

</comment><comment><USER14> As well as patents, there is another class of intellectual property rights applicable to plants called breeders rights: <http://en.wikipedia.org/wiki/Plant_breeders%27_rights>.

</comment><comment><USER15> Plant patents are just for clones. If you gre it from a seed, it is not covered by a plant patent.

</comment><comment><USER16> Roses are also patented, so you're not supposed to propagate them by planting cuttings.

</comment><comment><USER17> you can patent A FUCKING PLANT??? americans are batshit crazy if you ask me

<comment><USER18> Not sure where you live, but legally in the EU it is even worse, though thankfully orders of magnitude smaller scale than the US at this time.

The EU directive "on the legal protection of biotechnological inventions" (Directive 98/44/EC), implemented by all EU member states and even some non-EU states as of 15 January 2007, states _"biological material which is isolated from its natural environment ... may be the subject of an invention even if it previously occurred in nature."_!

This goes even beyond an international agreement on IP rights (TRIPS). [1]

The main mitigating factor is that EU governments are not particularly IP conscious and are terrified of the public backlash if they attempt to enforce any bio patents - most Europeans have no idea DNA is patentable! In addition, most of the infringers are actually related to the state due to very large welfare systems or academia. Private companies do not have a strong or irreversible grip in these areas, yet.

[1] [http://en.wikipedia.org/wiki/Directive_on_the_legal_protecti...](http://en.wikipedia.org/wiki/Directive_on_the_legal_protection_of_biotechnological_inventions)

TL;DR. Everywhere is batshit crazy these days...

</comment></comment><comment><USER19> More IP craziness.</comment>


The Pink Floyd Night School - <AUTHOR> http://www.nytimes.com/2010/05/02/opinion/02edmundson.html?hp=&pagewanted=all <comment><USER2> That is, perhaps, the best pre-plug for a book disguised as an article, I've ever read.

<comment><USER3> Nonetheless, it _was_ really good.

<comment><USER2> True 'dat.

</comment></comment></comment><comment><USER4> It doesn't seem to be quite as easy to get a job as a tenured English professor these days as it was for this guy. Maybe that's why our generation is more nervous.

<comment><USER5> Who says it was easy?

</comment></comment><comment><USER6> It reads as if Stevie Nicks had been part of Pink Floyd. Wonder how Prof. Edmundson grades such paragraphs.

It should be said that it was a lot easier to graduate from college with little or no debt in those days. Private college tuition for the 1973-1974 school year was probably under $3000. One could often find work in the summer (not coding interships, but manual labor) that would bring in $1200 or $1500.

</comment><comment><USER7> What an arrogant ass, gloating about having a rich enough family to have his back that he got to play at being poor while between colleges.

</comment><comment><USER8> It's the Floyd, THE FLOYD, not Pink, she's still a Rockstar.

<comment><USER9> Apparently - Pink isn't well, he stayed back at the hotel.</comment></comment>


San Diego: join us for Hacker News meetup #16 (Fri 4/29) - <AUTHOR> http://anyvite.com/wz0ntoug7c

<comment><AUTHOR> Please RSVP! We typically have 20+ people in attendance, so to set aside room for us at the coffee shop, I'd like to have an accurate headcount.

This is a purely social meetup event, and is not based around any sort of technical presentation. It brings together a unique blend of technology and entrepreneurship discussion, and it's an opportunity to interact with this special community in real life.

To receive notifications of future events, join our e-mail list: <http://groups.google.com/group/sd-hackernews>

More info on the homepage/wiki: [http://cses.ucsd.edu/wiki/index.php/San_Diego_Hacker_News_Me...](http://cses.ucsd.edu/wiki/index.php/San_Diego_Hacker_News_Meetup)

For other San Diego area tech events, see the huge SD Tech Scene calendar: <http://www.sdtechscene.org/></comment>


The first mile problem in Fintech - <AUTHOR> https://medium.com/@charlestm/the-first-mile-of-fintech-b48db8b21897 <comment><USER2> "a critical piece will be solving the first mile problem by making sure that we lower the cost and the effort required of all of us when it comes to transitioning into new payments and monetary systems."

This. Right now the costs to start a fintech company that actually does something disruptive is tremendously high and the regulatory hurdles don't stop. Luckily there are a few startups paving the way but it's still not enough.

<comment><AUTHOR> I know I'm biased but I really and truly believe we're helping solve this. We remove the need to build out ACH processing with a bank, do the regulatory work for you, and roll in a pretty cool account ownership verification piece out of the box. No ODFI needed, no Yodlee needed, plus if you want we'll guarantee the funds so you don't even need to take float risk or monitor returns. We do it pretty damn cheap.

I'm writing this not so much to make a pitch (of course that's part of it) but to point out what I hope other fintech companies do: package UX and compliance together and sell it for a reasonable cost. It will help the whole industry explode into the potential investors are clearly seeing in it.

</comment></comment><comment><USER3> The article offers a promising future for "FinTech" without giving any reasons _why_. Paypal, started in 1998 _was_ the promising new future, but that future has fallen flat. Their list of prohibited businesses[1] reads almost the same as Stripe's[2], which, for a business that is trying to fix the pain in 'online payment processing' is shameful, but business as usual. (It also doesn't do a great job of defining its usage of 'first mile' vs 'last mile'.)

Is there anything that has changed that changes the legal landscape which would allow for any innovation? The SEC announced that it was considering change to the standards for being an 'accredited investor' but as far as I'm aware, ultimate didn't go anywhere. This would have made crowdfunding platforms much more tantalizing but didn't seem like it would have affected the larger marketplace.

Money laundering is very much still a crime, and shows no chance of changing anytime soon, but the requirements borne out of that are onerous (KYC) and most banks are not going to keep an account that has large sums of money going in and out of it open for very long.

The problem with finance was never the technology (Western Union's person-to- person money transfer service dates back to 1871!) but always the regulation, which, again, shows no evidence that anything will change.

[1][https://www.paypal.com/us/webapps/mpp/ua/acceptableuse- full](https://www.paypal.com/us/webapps/mpp/ua/acceptableuse-full) [2][https://stripe.com/us/prohibited- businesses](https://stripe.com/us/prohibited-businesses)

<comment><AUTHOR> The primary reasons people believe Fintech is primed for explosion are the following:

1\. Ubiquity and mobility of the internet. Paypal was designed for when the internet was new and barely anyone was using it. Now, everyone is on the internet and on it constantly with devices that have way more powerful. Now is the time for just about any innovation that was promised but not fulfilled in the first dot com bust to have another shot.

2\. Stirrings all around - talk to any bank and ask them if they think financial technology is about to break out. They're pretty much all convinced it is because they see it pecking at the edges. Alternate lending just has 2 IPOs 10 days apart (on deck, lending club), crypto currencies are surprisingly stable given what they are and how new they are, and new payment tech is gaining traction (apple pay, google wallet, square, venmo, etc). It all starts small but it's gaining traction faster than it used to, and momentum is everything.

That's what I've seen and what I've heard as the two biggest reasons. Essentially, reach and momentum.

</comment></comment><comment><USER4> Hey all, thanks for taking a look at my article. Would love to hear your ideas on closing the first mile gap.

<comment><USER5> * Solve at least one problem ordinary people actually have.

e.g. Bitcoin conspicuously fails at this. There is nothing the typical consumer can do with it that they can't already do with credit cards or PayPal, and those offer chargebacks and rollbacks in case of problems. Even remittances turn out to cost more in practice.

What can you do that existing centralised systems don't already do? And remember that Shit Happens, and rollback is considered an essential function by the trepidatious.

</comment></comment><comment><USER6> Utterly pointless article.

<comment><AUTHOR> Utterly pointless comment</comment></comment>