Ask HN: Anyone interested in building tools for showing bias in news? - <AUTHOR> We all know how the election turned out. I believe the internet is contributing to echo chambers because of filter bubbles. I think it is rare to see some news that is not biased one way or another, and I believe it is hard to expect it not to be so, because of our own individual biases. I think there is a need for tools that can help us see the other side.
What if we had a tool that is delivered as a browser extension, that can show links to alternate views (think URL links) of the exact same topic that you are reading? It can use information like how biased the current article is towards different people/events/ideas and find an alternate article that can help you understand other-side. It may not solve all the problems but it would be a good start. Would you find it useful? Do you have a better idea?
This requires skills (NLP/ML) that I don't have but I am willing to spend time/effort (I am a programmer/big-data-engineer) to make it a reality. Would anyone be interested in working on it? It will be open source and any organization that runs this will be non-profit. <comment><USER2> I'm concerned that your frame for this might be off as I don't think the "filter bubble" is rational opposing viewpoints around a particular topic so much as there just being so much misinformation and plain weirdness out there.

There's a saying about it being 10x harder to refute bullshit than it is to spew it. How do we in the information and technology wing of society build tools to deal with that?

Alex Jones had a rant about how Obama and Hillary Clinton both smell like sulfur because they're demons.

I'd assert it's a "real story" and exactly the kind of filter bubble issue we're talking about as Alex Jones was personally thanked post election by Trump [1] and when it happened the sitting president of the United States made remarks about it [2].

I had a real conversation with an elderly relative of mine who told me quite straight faced that they read all about this and how it was true - this isn't bubbles it's different realities.

1 - [http://www.huffingtonpost.com/entry/donald-trump-alex- jones_...](http://www.huffingtonpost.com/entry/donald-trump-alex- jones_us_5829eb21e4b0c4b63b0d9249)

2 - [http://www.cnn.com/2016/10/12/politics/obama-sulfur-smell- al...](http://www.cnn.com/2016/10/12/politics/obama-sulfur-smell-alex-jones/)

<comment><USER3> Indeed, if we do not agree to a common set of facts and a shared reality, how then do we proceed with any kind of meaningful debate? To paint a basic picture, how do you debate the feasibility and merits of sending humans to Mars with a person who denies the moon landings? You might say these are extreme cases, but with the rise and "normalization" of Alex Jones-type media outlets, this will only get worse.

<comment><USER4> Exactly, when it comes to reality not everybody gets a trophy. Critical thinking and science are waaaay undervalued in our culture.

<comment><USER5> Reality is lot of the science is wrong as well. Intentionally paid by special interest or the end result is defined before any science is done.

You don't want to look science religiously either.

<comment><USER6> IIRC one of the main tenants of science, when I was learning it in middle school in the early '90s, was that it's fact until it's disproven. We don't know everything about it which is what makes it different than religion. There's no faith involved.

I know a lot of it is corrupt and aiming for a target but good scientists will tell you that they're basing their findings on studies which may have flaws and they'd want you to try and reproduce their studies to tell them if they've gotten something wrong.

<comment><USER7> This is what falsifiability is all about in science.

Mathematics relies upon proofs, but there is no such thing as a scientific proof.. You can find sufficient conditions for a phenomenon and have a wholly water-tight argument as to why it occurs, but you can't use use observation to find necessary condition.

This is because observations are evidence, and if a future observation shows problems with your theory, you should not discard your evidence (unless of course your secondary theory is that your evidence exhibits a model for some reason why it doesn't explain what it seems to explain on the surface, e.g. experimental error), yada yada..

<comment><USER8> I'm not sure this is True, as you can use probabilistic arguments.

<comment><USER7> Probabilistic models are what I was referring to; all probabilistic models should produce a prediction, and also a measure of confidence. That measure of confidence can never be 100%. Similarly, you can never have an incontrovertible belief that is not a mathematically/logically-constructed fact 'outside' of the messy real world in some abstract chamber walled off from anything that is like a real life observation, no matter how certain. Basically, you can't have P(A) = 0 or 1 as your prior probability, because then you break Bayes' rule in the sense that new evidence is meaningless - [https://en.wikipedia.org/wiki/Cromwell%27s_rule](https://en.wikipedia.org/wiki/Cromwell%27s_rule)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I think the tool you need is "skepticism". Teach people to not rush to ascribe labels to everything they don't like, and to stop ignoring anything that has been labeled as X-ist by their bubble. Teach people to not think with slogans and hashtags or listen to celebrities.

<comment><AUTHOR> Well, that may work but you also need to have intellectual resources to be able to process all the information and see something for what it really is. Also, people serving news are upping the game by using the latest in psychology research to influence us. We need _somebody_ on our side.

<comment><USER9> Hmm, i think its a very difficult task. I think perhaps a useful thing would be to intentionally break the bubbles. Something like an inverse- recommendation engine. E.g. instead of "people who read article X also read Y", you say "people who read X did not read Z". Or if you had access to people's facebook likes (does facebook provide an API?) you could find the most anti-correlated users and recommend each other's liked stuff etc.

<comment><AUTHOR> Isn't what I mentioned sort of similar? It is difficult for sure which is why I am asking for help.

<comment><USER9> I think people in general avoid the other side, and they don't have enough time for that. Thats why i suggest a mix up of sides. That way you read each story once, but as a whole you get exposed to both contexts.

<comment><AUTHOR> Interesting. I'll keep this in mind.

</comment></comment></comment></comment></comment></comment><comment><USER10> > I think it is rare to see some news that is not biased one way or another

I would posit that part of the problem is the implicit assumption that biased != factual. This assumption is true on both sides of the political spectrum, but it takes on a different character for those on the right.

I understand the sentiment that prompted you to post this, I'm just not sure what you propose is any kind of a solution. If anything I believe it may make things worse by affirming the bias != factual assumption.

<comment><USER3> I would also add that the "bias argument" is trotted out so much that it appears to be turning into a cliche in its own right. For instance, I may link to a cogent, factual article published by Politifact only to have it quickly dismissed as a "biased" source by those too lazy to consider its content.

</comment></comment><comment><USER11> This is a pretty good website and they also have a proprietary algorithm to determine the political spectrum of the source: [http://www.allsides.com/](http://www.allsides.com/)

NYTimes is ranked moderately liberal while Fox News is ranked right ([http://www.allsides.com/bias/bias- ratings](http://www.allsides.com/bias/bias-ratings))

<comment><AUTHOR> Thanks! It seems interesting. The problem I see is that - it's hard to pull people away from their news sources to something unbiased. We need to take unbiased articles to the people. Also, journals shouldn't be the only source. Every side needs to be heard and someone's voice may come from their own blog. I am not sure how it would work but we should at least give it a try.

<comment><USER12> I'm curious how you think this would work.

First of all, I'm interested in what you think a news publication free of bias would look like. Would it be a straight recitation of factual occurrences? That would be useless, right? How would you decide which story gets a big headline? Which detail gets included?

Second, the USA Right in general has completely broken with the idea that there is any such thing as "fact" that matters. Evidence, etc. is irrelevant. Look at the Right's views on climate change, etc. I'm sure there are people on Hacker News etc. who will be like "now you're just being a Bias Liberal!" if I don't say "the Left has its problems too" but the Left's problems, such as they are, are the traditional problems of political parties: maybe the Left's policy prescriptions won't work, maybe their need to maintain a coalition means they aren't able to address Problem P, etc.

The Right in the USA is unique in having decided that there is no evidentiary standard high enough when Rightist positions conflict with reality. So removing bias from news sources and ensuring everything is quite factual won't affect the ~40% or so of the USA who frankly couldn't give a flying fuck about reality.

<comment><AUTHOR> I am not interested in unbiased news sources. I am interested in showing alternately biased news source so that you can develop your own opinion. Even showing snippets of alternate views superposed on the same article can help people a bit.

My aim is not to solve the problem in its entirety. If others cannot understand our opinion, we should at least try to understand theirs. The current situation is horrible in terms of people not even having a clue as to what's going on. If 10% benefit from this, that means there are 10% of people who at least know both/many sides.

<comment><USER13> > it's hard to pull people away from their news sources to something unbiased. > We need to take unbiased articles to the people.

> I am not interested in unbiased news sources.

So, which is it? I think the focus on bias vs unbiased news sources is a distraction. ALL news is biased. That is something that will never change.

> If others cannot understand our opinion

What if the issue here is that others understand your opinion perfectly, but still disagree with it? This whole conversation is starting to sound a lot like you want a tool that will magically make people agree with you.

<comment><AUTHOR> Good catch regarding the biased/unbiased. Parent to that post pointed out that there is a news source that tries to show unbiased news. I was implying that we need to take such news sources to people, rather than expecting them to get to that on their own. As I mentioned elsewhere, the idea is not to show unbiased news but rather show other side of the news (in the bias spectrum). I am not entirely clear on what makes the most sense, which is why I posted it here.

I don't have any claims on people not agreeing with me. I have claims on me not understanding what's going on. I think if such a tool as I proposed existed, I would be happy to use it.

<comment><USER14> The best "tool" available is the No Agenda Podcast.

Hosts Adam and John present national and global issues as portrayed by mass media, and systematically DECONSTRUCT the stories. They not only highlight biases, but explore propagandistic elements and discuss how the media intentionally affects the consumers.

Most importantly, they work to identify the MOTIVATIONS for medias' biases in the first place, and demonstrate for their listeners how to be much more critical, skeptical, and analytical in the way they consume the news.

Listen to a few episodes and you might rethink the need for building a tool at all.

<comment><USER15> In all aspects, they would be brilliant, but what about spectrum of content coverage. What about readership?

But this dissemination of information by identifying biases of the entities involved is a tactic that can be used by anyone. Objective thinking belongs to everyone, doesn't it?

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER16> I also don't have the skills for building such a thing. But if I could help with such a project through testing/QA, or contributing to a website, or gathering resources - whatever, really - I would, and I would use it. I've been thinking a lot about confirmation bias and echo chambers lately, too, and the truth is I don't know exactly where to look for opposing viewpoints all the time, and it's hard to gather the courage to just dive in when I know a good amount of it will make me feel ill.

Maybe just starting with a collection of opposing resources? There could be a call to action to ask people to submit articles or sources for various "stances" on different topics, and a list divided by topics and view points, or links to the few sane and awesome discussions you can occasionally find where people who think differently actually talk to each other about their differences like rational human beings. That might be an approachable place to start, anyway.

<comment><AUTHOR> I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation.

</comment></comment><comment><USER17> One component of this problem that is rarely mentioned is bias-by-omission. Journalists commit this sin frequently and vigorously. With enough elbow grease, you could make some progress towards detecting it.

</comment><comment><USER18> It is certainly an interesting idea, and I don't want to discourage you, but how would you reach the people who would benefit from it the most? If you are aware of a filter bubble, you can easily seek out alternative information...

<comment><AUTHOR> Good point. If it works, maybe we can convince browser developers to include this as default similar to how security is done. We are putting more effort to secure our computers than securing our minds. Even if I were aware of being in a filter bubble, it takes time and effort to get out of that. We need to make it easy.

<comment><USER19> I agree with the comment below about this being a technological solution to a human problem. That said, I'm interested in looking into the NLP/data side of things, and I look forward to how this idea would supplement existing news sources. Keep me posted on the status of this project! :)

<comment><AUTHOR> I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) \- drop me an email at my id @ gmail if you want to join the discussions

</comment></comment></comment></comment><comment><USER20> I saw two big issues on both right wing and left wing political blogs and news sources over the election: assumption of bad faith (and in the worst case intentionally taking the least favorable interpretation of some news item) and lack of fact checking. Untruths abound.

It happened on both sides, people were routinely taking everything Trump was saying and turning into a joke or making it out worse than it was (some things were very bad and deserved the attention, but lots of things weren't bad but were painted with that same brush). Until people are willing to admit that whatever the result will be of this kind of project will not address the root of the problem.

</comment><comment><USER21> I think the idea is promising but I'm pessimistic that those who need it most would opt-in.

It sounds like a valuable project: good luck.

<comment><AUTHOR> Thanks. It is clear that most organizations don't want to invest effort in doing what is right. If there is some ready-to-use extension/service that is proven to help, maybe we can convince the browser developers to include it in their default installation.

<comment><USER13> That's a dangerous idea. You seem convinced that you're right and other people are wrong. So much so that you want to force this idea of right and wrong on people by installing it in their browser by default?! This smacks of soviet era re-education.

> maybe we can convince the browser developers to include it in their default > installation.

Any browser that decided to do this would quickly find themselves out of business. Free will exists for a reason. Attempt to circumvent it at your own peril.

<comment><AUTHOR> I am not sure if we are talking about the same thing. I am not talking about right and wrong. I am talking about someone's opinion and alternate opinions.

<comment><USER13> It's the "let's add it to the browser by default" part I have an issue with. If people want to be able to see alternate opinions easily, they'll install it on their own. If they don't, no amount of forcing it on them will get them to consider alternate opinions. It'll just piss them off.

</comment></comment></comment></comment></comment><comment><USER22> Another issue is that many biases are structural, technological or driven by commercial pressures.

For instance, I would say CNN is biased toward coverage of school shootings and airplane crashes. CNN has the problem that there is not enough news to fill 24 hours so they run a heavy rotation of the same crap that is cheap to produce. Probably the best footage they show is stuff they downloaded off Youtube.

When you catch the CNN crew on a slow news Sunday they will admit that their problem is engaging an audience, both in the sense that they need to make money and also in the sense that they have some duty to inform the populace, the populace has duty to inform itself, etc. The truth is their content is boring, depressing, and awful but they have varied their formula a lot and they really believe they've found a local maximum of what people will watch.

In some sense CNN was biased towards Trump because he's interesting. I would look for news about Trump every day because it was likely he would say something crazy again and I think this was the case for a lot of other people. CNN, Fox News and MSNBC all had great ratings this season.

This 1971 book

[https://www.amazon.com/Information-Machines-Ben-H- Bagdikian/...](https://www.amazon.com/Information-Machines-Ben-H- Bagdikian/dp/0060902582/)

is about as ahead of it's time as Ted Nelson's work and is very much about what news would be like in the age of the World Wide Web and it contains a damning indictment of the very concept of "news". (i.e. not only is there not enough news to fill a 24 hour tv show, but it's arguable that there is enough news to fill a newspaper every day)

<comment><USER23> >In some sense CNN was biased towards Trump because he's interesting.

Bias with respect to how much coverage there was of the candidates, however many would say the content of that coverage was biased against him.

Besides this small point I agree. Time spent on a particular subject or topic is a subtle bias in itself which can be driven by many things (including money) and has huge effects on public perception of people and issues.

How does one expose bias in this subtle behaviour in an automated way, even with machine learning?

<comment><USER22> I think the sheer amount of press coverage that Trump got made him stand out of the pack.

Really I think a lot of the negative coverage Trump got could have been beneficial to him. For instance, if you are looking for somebody who is going to "shake up Washington", then denunciations from Republicans such as Mitt Romney and Paul Ryan add to his credibility.

<comment><USER23> I think it's more probable than not that it ended up helping him, but it significantly polarized opinion of him as a candidate.

</comment></comment></comment></comment><comment><USER24> I recently began working on a (somewhat) related NLP project looking at the shift in sentiment in Trump-focused articles published pre- and post- election. The motivation for the project is the observation that many of the left-leaning news media outlets - who consistently lambasted Trump in the lead-up to the US election - have begun to dial back their criticism post election, presumably in an attempt to (re)build bridges and ensure their continued relevance in the ensuing Trump milieu. It seems that the results of such an analysis have the potential to be a concrete and relatively simple example of the deviation from any stable media narrative, and perhaps a nice opportunity to spread this message to a slightly wider audience.

The project is nascent, but it should be straightforward to implement (I have already begun to amass articles from several major news sources). While this may or may not be relevant to your stated goal, I'd be happy to share more info if you're interested!

<comment><AUTHOR> Hey pinto, I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation. I would like to make use of your code for fetching articles from different news sources as a starting point. You are free to contribute to the project as well. Please let me know!

</comment></comment><comment><USER25> What I've wanted for a while was a curated news source that allowed me to collapse topics (e.g. only one story about the presidential race per day). The goal would be to present both "perspectives" as well as identify the core shared truths between the two.

The problem is cross-article context comparison is actually a bit harder than news article summarization and the amount of time required to pursue it made it seem a bit too much of a chore.

One extension would fall to politicians and public entities that make statements where it could validate/compare their statements to their historic actions. Beyond the "is this reporting accurate" it would go into "do we think this actor is being truthful based on historic behavior"

Edit: The other nice thing about this is that I could hear about the things that _aren 't_ the recent election cycle or terrorist attack. It's like sensationalist news signals were saturated which raised the noise floor drowning out all of the other news.

</comment><comment><USER26> We have a small team of researcher on that topicm working from Berlin and Paris, come say Hi or get in touch ! :) [http://cmb.huma-num.fr/wp- content/uploads/2016/02/Algodiv-Ge...](http://cmb.huma-num.fr/wp- content/uploads/2016/02/Algodiv-GeneralPresentation-2016-en.pdf) A great article of O'Reilly about this topic too : [https://medium.com/@timoreilly/media-in-the-age-of- algorithm...](https://medium.com/@timoreilly/media-in-the-age-of- algorithms-63e80b9b0a73)

<comment><AUTHOR> Hey mazr, I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation. I think your expertise could be of real help in building this.

</comment></comment><comment><USER27> I'm already working on my startup which started way before this election. News bias didnt happen and it is not a new problem. It's been there for centuries. Unfortunately, it's my startup idea and I don't want to open source it. The work is in its super early stages but if some one is interested in teaming up on this and working together, shoot me an email at email@example.com. I live in Silicon Valley so if your local it will be great but location doesn't really matter if you passionate about the idea!

</comment><comment><USER28> Instead of making some sort of judgement call (this is biased in X manner), how about a tool along the lines of something I read about in an SF story a long time ago (sorry, don't recall the title or author). I remember the point- of-view character looking at news items on a screen and things like "connotation indices" and "hyperbole metrics" were included. Those sorts of indicators would give each reader a chance to realize to what degree bias exists and leave it to them to decide how important that was.

</comment><comment><USER29> I was on an Assembly project that tried to do this, think it was called "Flipside". It never went anywhere, IIRC required coders didn't materialize.

Having been inspired by sama's dialogue regarding the downside of unfriending those with opposing views (on the election), I've militantly kept up on opposing Facebook friends' perspectives, giving conscious effort to see their point(s). I see the value in adding opposing news sources to my feed, but the rancor I see (on both sides) is a turn off. Haven't found reliable opposing sources that don't require that I, at least at some level, apply a sort of what I've come to refer to as 'normalizing' their points. So much emotionally charged rhetoric- I guess the 'sizzle' factor sells, but requires additional calories burnt to see through & try not to be disproportionally influenced by.

Maybe a sub-Reddit or sub-Voat -type thing could be built which includes meta- rating elements to allow for rating bias leanings. Dunno what kind of software might already exist that could do this kind of thing for cheap.

tl;dr: A failed Assembly project tried this recently It's hard to create a fair Facebook feed of opposing views A Reddit/Voat -type board with meta- elements to track bias might exist cheap

<comment><USER15> I'm thinking on the same lines as you! :)

</comment></comment><comment><USER30> A different or supplementary approach could be removing the bias from a news item.

Instead of trying to determine which biased side an article is skewed towards _and_ then finding other links to what is determined to be the "alternative view". Scrubbing bias or at least highlighting it is already helpful.

For example: at work we get daily emailed briefs with major business news items summarized to ~3-5 bullet points of facts. Journalist opinions/bias and rhetoric language is mostly removed in the bullet point sentences. It's not a perfect system by any means, not even close, and I would love to see something similar offered that's improved and expanded in what it can do.

This type of bias scrubbing/summarizing is easier in business news and sports news which involve more numbers and figures reporting (+nowadays many of the full articles may also be written entirely by bots - see link below). It would be harder to expand this for longer investigative/politics news articles. But a partial imperfect solution here is better than status quo.

I would be a user of a tool that could summarize key "unbiased facts" from articles and I would be interested in helping build it too.

Link to a NYT story about algos writing/summarizing news: [http://www.nytimes.com/2015/03/08/opinion/sunday/if-an- algor...](http://www.nytimes.com/2015/03/08/opinion/sunday/if-an-algorithm- wrote-this-how-would-you-even-know.html?_r=0)

<comment><USER31> What if the bias was created by omitting important facts that, for example, contradict the narrative of the item.

Sports are indeed a good domain for what you're talking about, but that's because they're run by well known rules, it's a highly artificial domain, not directly including the messiness of general human affairs (except in that they're played, refereed, managed etc. by humans).

Business ... well, how many stories about Yahoo! have ignored its negative market net worth when its Alibaba stake is removed, when that was relevant? Not necessarily examples of bias, but....

<comment><USER30> Good point about omission of important facts. It's tough because facts can be absent from a news item intentionally and unintentionally. Journalists have time and size constraints - so even when striving for accuracy and objectivity, facts may just be missed and some other known facts will certainly be left out as determined not relevant enough at the time of writing. This is just the reality of reporting news.

Professional Journalism has ethics and standards - of course w/ ongoing debates in how they are practiced - and they are taken seriously by the industry as a whole. Wikipedia has more info: [https://en.wikipedia.org/wiki/Journalism_ethics_and_standard...](https://en.wikipedia.org/wiki/Journalism_ethics_and_standards)

Ultimately, for news items where a reader wants to get a fuller picture of the story he/she is going to have to go read multiple longer articles about it. Scrubbed bullet points may offer discovery with less bias at this initial stage and from which to choose those stories to go read more about.

I don't want to get off topic, but the Yahoo example mentioned is interesting and I think worth maybe discussing so

 _TL;DR skip this next part if very wonky Yahoo stock news reporting is not an interest:_

Where did you get your Yahoo business news?...Firstly, Yahoo's (YHOO) value is not negative ex Alibaba (BABA). "A company's shares are worth what someone is willing to pay for them" is a relevant quote here and Verizon offered ~$4.8 Billion to buy Yahoo ex Alibaba (also ex Yahoo Japan btw).

Pretty big difference between a negative value and $4.8Billion!

Second, if talking about a stub Yahoo valuation? Like using some shorthand value estimation method based on looking at: (BABA's recent stock prices)x(size of Yahoo's stake) and comparing that amount to where YHOO's market cap is trading at? Then yes you could end up at all kinds of crazy totals, including negative numbers, depending on what shorthand valuation method you chose to use...

But these types of calculations, maybe interesting and fun, are basically meaningless in the real world. The amount Yahoo will receive for its Alibaba stake at some unknown time in the future cannot be calculated accurately from BABA's stock price today. Nor can it derived from whatever $ amount Yahoo has written in its books for the value of their BABA stake (Book value <> market value). (Sum of parts also <> market value)

Yahoo can be trading at a discount (or a premium) to one's hypothetical future value calculations for all kinds of reasons: eg YHOO's potential +$10B tax liability uncertainty should they sell their BABA shares with capital gains (Yahoo still talking to IRS about this and its unclear how it will play out but it's shocking Yahoo has not been given clearer answers by the IRS yet to quite fair questions about the tax code)

Yahoo is also still facing material issues in Verizon deal not yet finalized (YHOO email breach may change terms, this is also unclear), etc.. etc..

Basically, _Stock Valuation Is Not News_. Though some stock trading websites look like news or are attached to legit news sites (Barron's to WSJ for example). Barron's is not news.

NYTimes DealBook discussed Yahoo stub being negative. DealBook is a blog. [http://www.nytimes.com/2015/12/10/business/dealbook/how- to-v...](http://www.nytimes.com/2015/12/10/business/dealbook/how-to-value- yahoos-core-business.html?_r=0)

An analogy for those not familiar with Yahoo's stock price valuation: This is like guessing how much money you could receive from selling your house if you tore it down and sold it for scrap wood and separately put the plot of land on the market. You also have to calculate this future value without knowing if your local council will approve the idea, or how long it will take, or what prices for scrap wood/plots of land will be, or how much tearing down _your_ house will cost, or the % odds you secretly have been living in a house built over an old unicorn graveyard and can't sell the land at all.

One can imagine all these factors are why a number calculated for future "value" of stock may be _very_ different than what actually is received when a sale happens. Like way off. Like a negative number vs. +$4.8Billion way off.

</comment></comment></comment><comment><USER32> I had thought about something like this a couple of years ago. My idea wasn't really about tracking bias, but providing reputation for authors and news outlets. Users (via a browser plugin or some other mechanism) could declare an article or statement as biased in a certain direction. That vote would give me information about the article, the news outlet and the reader. I could then present that information back to users. In this way I could learn just how biased I am (and maybe even the areas of my bias) and the bias of authors and organizations. Perhaps you could see how an author's bias changes when they write for different sites.

However I don't think just pointing out bias will really help. People like their bubbles, and moving out of them is painful and potentially with real world consequences for them. I also think if you show a user an articles bias ahead of time, it will just be used as a filter or a way to reinforce their bubble. I thought this article was interesting.

[http://www.vox.com/2014/4/6/5556462/brain-dead-how- politics-...](http://www.vox.com/2014/4/6/5556462/brain-dead-how-politics- makes-us-stupid)

<comment><USER15> Very insightful.

</comment></comment><comment><USER33> How do separate bias from truth?

If I said more asian people are smarter than white people I am biased and racist.

If I said more asians have black hair than white people I am not biased, I'm stating an objective fact.

The only fundamental difference between the two statements is that there are hard numbers lending support to one statement (asians having black hair) and the other statement does not. Neither statement, from a technical standpoint, can be verified definitively.

To build a machine that identifies whether or not a statement is biased one must first build a machine that identifies whether or not the underlying statement is true or false.

Building such a machine is an impossible endeavor because the means in which we identify whether or not something true or false is through data, a source which in itself can be biased.

</comment><comment><USER34> I built a search engine for lectures ([https://www.findlectures.com](https://www.findlectures.com)), partly to explore this problem. I think what you're describing would be challenging, but if you do work on this please feel free to contact me if you want to discuss ideas.

My approach has been to obtain collections based on recommendations, filter out low quality material (bad audio, lots of ums, etc) and categorize it so you can explore freely.

Someone has to fund free content, and the missing topics are a form of bias - I can't guarantee that there is a counter-argument to every lecture.

</comment><comment><USER35> Great idea! I am working on pushing my Facebook colleagues to work on this as well. I think the problem will not be solved unless Facebook makes a serious effort. The issue is that like the filter bubbles that are created in the social network, the company is also in an echo chamber of its own.

So, I think for all the folks that know Facebook employees, break them out of their bubble (if they are already broken, commend them and encourage them to improve Facebook). Show them that Facebook is not so rosy colored as it claims, and that they have a responsibility to build tools that promote truth and inclusion.

<comment><AUTHOR> I think it would gain much more credibility if this were done in the open in a transparent manner. If there is a perfect fit for open source software, this is it. If folks from facebook would like to invest time/effort that would be great!

</comment></comment><comment><USER36> Dude... people dont even believe in the scientific method anymore, and definitions and facts are themselves liberal bias. What is some tool gonna do to convince those who have shut down entirely??

</comment><comment><USER37> i've been interested in something similar for quite a while, although my conception of it was more along the lines of a browser extension which would identify weasel words, unsourced statements, and alert the user about "experts" who are biased.

i'd be interested in being an editor/philsoopher for a project like this, given that someone would need to determine what counts as bias and what doesn't.

i do think that apps to improve critical thinking are direly needed...

</comment><comment><USER38> People would believe what they want to believe about your assessment of bias just as they already believe what they want to believe in the news.

Caring about truth has to come from within.

</comment><comment><USER39> What about a tool that allows you to find and start a chat with someone currently reading a differently biased account of the same (or similar) events? I can't count the number of times this election that I wished I could discuss a Times article with a Trump supporter; I am sure there are plenty who wish they could tell me about the contents of a Breitbart article.

</comment><comment><USER40> I think it would be best to start with a small feature. For example, build a classification tool seeing if perspectives of both sides of an issue are interviewed and quoted. Open source it along with your dataset.

We need to build more composable machine learning based tools, and then we can use machine learning on their results to determine which serve which purposes best.

</comment><comment><USER41> Another suggestion - a clear description of both author and publisher funding and political links which are likely to lead to bias.

</comment><comment><USER4> You might want to monitor the progress unfolding on this list of dubious news sites. It seems relevant to your endeavor.

[https://docs.google.com/document/d/10eA5-mCZLSS4MQY5QGb5ewC3...](https://docs.google.com/document/d/10eA5-mCZLSS4MQY5QGb5ewC3VAL6pLkT53V_81ZyitM/mobilebasic)

</comment><comment><USER42> Related to this, have you seen the Wall Street Journals red/blue feeds, straight from the Facebook API?

[http://graphics.wsj.com/blue-feed-red-feed/](http://graphics.wsj.com/blue- feed-red-feed/)

Just seeing articles on the extreme side of each topic right next to each other shocked me.

</comment><comment><USER43> Unfortunately it often appears that one person's bias is another's fodder. Dailymail.com is incredibly astute and manipulative—that's their business. Bias? Of course, and they know it. They would tell you, I suspect, that they are in the bias business.

Is there a solution? A biasometer?

</comment><comment><USER44> Not quite the same thing but Rbutr ([http://www.rbutr.com/](http://www.rbutr.com/)) was features on HN a while back.

In their own words:

"rbutr tells you when the webpage you are viewing has been disputed, rebutted or contradicted elsewhere on the internet."

</comment><comment><USER45> A while ago I wrote this about the subject. Maybe you would like to read it "The New Editors" @arisAlexis [https://medium.com/@arisAlexis/the-new- editors-122eeb57880d](https://medium.com/@arisAlexis/the-new- editors-122eeb57880d)

</comment><comment><USER46> Why is everyone suddenly convinced we can do this now? How do you guarantee your algorithm is without bias?

<comment><AUTHOR> We go with the assumption that everything is biased. The idea is only to surface the biases so you can make your own informed opinions.

</comment></comment><comment><USER47> Sorry for the offtopic, but I have to post this evergreen text:

>> There is voting, of course, but to become an informed voter all one needs to do is read a short guide about the candidates and issues before the election. There's no need to have to suffer through the daily back-and-forth of allegations and counter-allegations, of scurrilous lies and their refutations. Indeed, reading a voter's guide is much better: there's no recency bias (where you only remember the crimes reported in the past couple months), you get to hear both sides of the story after the investigation has died down, you can actually think about the issues instead of worrying about the politics.

Source: [http://www.aaronsw.com/weblog/hatethenews](http://www.aaronsw.com/weblog/hatethenews)

<comment><USER48> This just passes the buck to the voters guide. You still have issues of bias and misinformation. You may save a bit by being able to wait to see more information come in, but for some issues, you do want to know what's going on right now so you can act.

<comment><USER47> Did you read the post I linked in? It's says that the incoming information aka. news are irrelevant to our life and we should just ignore them. There are no real issues you have to act on. If a hurricane is coming, you will know it without reading the news.

Therefore bias and misinformation are also irrelevant to our life.

Just to have a counterpoint to this news obsessed discussion.

<comment><USER48> Yes, I did read the post you linked to. I am also responding to the quote you provided. I should think that you quoted it because you thought it was particularly relevant or representative, did you not?

Edited to add: I also found it frustrating that a post arguing against reading the news includes a "You should follow me on twitter here" link at the bottom.

 _Please don 't insinuate that someone hasn't read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that."_

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

<comment><USER47> >> I should think that you quoted it because you thought it was particularly relevant or representative, did you not?

And that's why I started it with "sorry for the offtopic". You do not always read what you want to but what you have to.

I just try to correct the bias I am concerned about. Lessons learned:

\- bias is subjective

\- there are people who like to stay in their own biased world

Am I relevant enough now?

Cheers!

<comment><USER48> I understood your "Sorry for the offtopic" to refer to your entire comment. I expected the quote to be relevant or representative with respect to the post _you_ linked to.

</comment></comment></comment></comment></comment></comment><comment><USER49> > What if we had a tool that is delivered as a browser extension, that can > show links to alternate views (think URL links) of the exact same topic that > you are reading?

The Pocket Browser Extension already does that.

</comment><comment><USER50> [http://www.newstrust.net/](http://www.newstrust.net/) when it was around was pretty good. It never really got traction though.

</comment><comment><USER51> Don't look for bias, just look for right or wrong.

<comment><USER52> Easier said then done. If this election cycle didn't showcase a post-truth condition I don't know what will.

</comment></comment><comment><USER53> SAAS to Facebook, etc. They can filter for low-bias news and help the zucc's PR.

</comment><comment><USER54> I would be interested (I was pondering on the same idea), how should we proceed?

<comment><AUTHOR> Hey huevosabio, I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation.

<comment><USER54> Thanks! I sent you an email.

</comment></comment></comment><comment><USER55> I don't know how much time I have but I would love to contribute somehow.

<comment><AUTHOR> Hey quantumhobbit, I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation.

</comment></comment><comment><USER56> harigov: very interested. you should setup a Discord channel to discuss further.

<comment><AUTHOR> Hey blipblop, I created a new GitHub project over here - [https://github.com/harigov/newsalyzer](https://github.com/harigov/newsalyzer) and a corresponding gitter chat group over here - [https://gitter.im/newsalyzer](https://gitter.im/newsalyzer). You can feel free to contact me through my id @ gmail so that I can start the conversation.

</comment></comment><comment><USER57> _I believe the internet is contributing to echo chambers because of filter bubbles._

People hear what they want to hear. If they are intentionally seeking out (or actively trying to create) echo chambers, you can't really stop that.

Additionally, studies consistently show that most people are pro choice. They also consistently show that most people are anti abortion.

In other words, no one is pro abortion. No one is " _for_ killing unborn babies." But some people frame the political policies in terms of rights of the mother to choose and some frame it in terms of rights of the unborn to live.

There is no easy answer here. Pretty much everyone thinks that abortion should be a last resort, not some kind of primary method of birth control. But there is enormous fighting about exactly where and how to draw the line on who, what, when, where and how.

So one of the problems you will find is that when you try to get "both sides" of _any_ argument (abortion is merely _one_ example here):

A) Either they are talking about very different foundational ideas such that it is kind of disingenuous to frame them as "opposing arguments" or

B) You have a set of people with such narrow views that they can only conceive of two possible options here and the real answer is to be looking for "a third way."

I think a better answer is to write about a broader point of view that helps promote a non-binary conversation and thought process and that helps promote that "third way" that has some hope of addressing real concerns for "both sides."

It's extremely problematic to try to divide more than 300 million people into two camps and pretend that the millions of individuals making up each "camp" all uniformly agree with each other. Yes, people choose sides in order to try to exercise power. But that doesn't mean the camps really authentically represent the full views and sets of positions of any of their members, much less all (of their members).

I think your desire to create this is rooted in good intentions, but I suspect that something like this will just help entrench the "war" by further promoting the idea that there are, in fact, two camps and only two camps and every American needs to choose one. I realize that is not what you desire to do, but that may well be the result.

I find that trying to have meaningful and nuanced discussion with almost anyone at all (other than my sons) is incredibly hard because most people want to peg me as either "for" what they are for or "against" them. This leaves no room for positing a third way at all, much less a potential fourth, fifth or Nth way.

I choose to blog as my small pebble of contribution towards trying to combat the either/or thinking and trying to posit new mental models for old problems.

If you do start something, I would be happy to give feedback or have some kind of limited (probably short term/one time) role in its development. I am not looking for an on-going time commitment. Furthermore, although I support your position that it "will be" open source, I see zero reason why this must be non-profit. I think this is just another example of common thought patterns that somehow being non-profit means it will be done for the right reasons, in the right way, etc and this is absolutely not in any way guaranteed.

</comment><comment><USER58> How do you actually do this without being biased yourself? One person's reality is another person's bias.

<comment><USER48> Similar to how science works. Build it with the fact that you have bias in mind, and keep your mind open to criticism, keep testing the system as best you can to test whether it's susceptible to any kind of bias you can think of, and remember it's always going to be an open question. Healthy skepticism. Keeping the work in the open is pretty important, too.

</comment></comment><comment><USER59> The supposed filter bubble theory is that I (and others) didn't vote for HRC because we are not exposed to the liberal point of view?

FALSE. LAUGHABLY FALSE. FALSE FALSE FALSE.

Let's make up a term called liberal privilege, the privilege to have your views reflected on all major US channels (CBS, ABC, NBC, NPR, BBC, NYT, WaPo... etc.) ALL OF THE TIME.</comment>


Customer Service: Old Technology is Often the Best - <AUTHOR> http://blogs.wsj.com/biztech/2008/10/06/customer-service-old-technology-is-often-the-best/ <comment><USER2> I agree with this. I am generally not a direct or assertive person, but when it comes to issues of customer service I strongly dislike the asynchronous aspect of most computer based communication.

Online chat wouldn't qualify as asynchronous, I suppose, but the one experience I had with it (ordering internal parts for a malfunctioning HP laptop) involved an agent replying once every 7-10 minutes -- one time it was so slow that I actually left for lunch before getting a response. Waiting 10 minutes to get a very clearly cut and paste answer does not give one happy customer service feelings.

So yeah, for me a quick phone call (even if I have to have the phone on mute for 10 minutes while I wait for a person) wins out over the new (cheaper) ways of communicating.</comment>


Fork() without exec() is dangerous in large programs - <AUTHOR> http://www.evanjones.ca/fork-is-dangerous.html <comment><USER2> In my opinion, it's libraries that secretly use threads under the covers (but still access state shared with the rest of the program like the malloc heap) that are what's dangerous.

<comment><USER3> Yes. Libraries should provide algorithms and data structures, and parameterize a threading interface if necessary (though I think it's usually not necessary).

Applications can then instantiate their own threads and pass them to the library (i.e. dependency injection.)

So if you have 10 different concurrent libraries in an application, then you don't end up with 10 different threading paradigms or 10 different thread pools. This is a matter of application architecture -- libraries shouldn't bake in a specific policy.

<comment><USER4> true...

...this applies not just to threading, but other things that are not in the domain of the library such as memory allocation strategy or communication mechanism.

Libraries should be small, focused and decoupled. This is the way to proper re-usability.

<comment><USER5> Like left-pad?

<comment><USER4> jeez, no.... extremes of everything are terrible.

I suppose another rule should be that there libraries should encapsulate a sufficient amount of complexity to make their existence worthwhile.... unlike left-pad.

</comment></comment></comment></comment></comment><comment><USER6> Yap. Erlang 19 (latest version) switched to using a smaller spawner executable and spawns all OS processes from there by forking. So it forks something restricted and small not the whole VM.

Here are the details of how it works:

[https://github.com/erlang/otp/blob/a5256e5221aff30f6d2cc7fab...](https://github.com/erlang/otp/blob/a5256e5221aff30f6d2cc7fab4875914ae734217/erts/emulator/sys/unix/erl_child_setup.c#L21-L49)

They also claim a 3-5x speedup for launching external commands because of it. So there is a nice performance boost as well.

So basically can add 4th strategy -- fork once a small program at the start, then fork from there from then on.

<comment><USER7> Thanks for pointing this out! Interesting that the code states that part of the concern is increasing memory usage. I don't quite understand why it would "duplicate the memory usage" since fork uses copy-on-write?

Anyway, I didn't explain it well, but this was what I was trying to get at with "fork a worker at the beginning of your program, before there can be other threads."

<comment><USER8> Copy-on-write doesn't work the same on every OS... Linux uses over-commit so it's not a big deal, but Solaris (and NT) don't over-commit so you actually need enough VM at the time when you call fork or fork will fail, so you may need to provide a lot of swap space on those systems to successfully call fork in large processes.

<comment><USER7> Ah thanks, that makes sense. Fun trade-offs: Either run out of memory when you call fork, or fight with the out-of-memory killer at a later time :)

</comment></comment></comment></comment><comment><USER9> In some programs, fork(2) is the right way to do concurrency. It's simple (API-wise), and it makes IPC explicit, as opposed to the implicit IPC of threads. clone isn't posix, so you can forget it, if you don't feel like being behind systemd in the "screw any system that isn't Linux" line.

In places where you need the speed, threads are useful, but they're harder to use than forks.

Annoyingly, threads and forks don't work well together. zzzcpan already talked about how to fork threaded code. As for the problem of libraries using threads, if a library you're using is using threads, and it's documented (it probably is), and you didn't know about it, I'll pencil that in as your fault.

Other comments in this thread have dismissed fork(2) as a bad job entirely, but I don't think I agree. It's an effective way to do simple multiprocessing, and it's a lot simpler than threads in many contexts.

</comment><comment><USER10> It sounds like if you use threads and fork it might be a disaster. Its true, but the author blames only fork and skip the second part of the problem, threads. This is not fair, because fork is much older than POSIX threads. In fact, POSIX threads were poorly designed to use with fork.

<comment><USER7> Fair point. My opinion is that in today's age of multi-core CPUs, shared memory concurrency is extremely useful for making efficient use of computing resources. As a result, I find threads to be unavoidable in most large systems I've dealt with recently.

<comment><USER10> Agree, that multi-threading probably the best choice for CPU-intensive applications. But for databases or http-servers multi-process + coroutines/fibers can be a better solution. For example, Redis is few-thread application. It creates additional threads only for disk I/O, because unfortunately file descriptors in Linux/UNIX do not work in async mode.

Btw, Redis reminds me, one really awesome usage of fork. To make a snapshot of itself Redis forks the main process. After that the child simply goes through tuples and write them to a files with out any worries that somebody will modify a records. Copy-on-write mechanism simply prevents it.

Redis save snapshot code: [https://github.com/antirez/redis/blob/unstable/src/rdb.c#L99...](https://github.com/antirez/redis/blob/unstable/src/rdb.c#L996)

</comment></comment></comment><comment><USER11> If you are in control of all the threads that an application is running, you can call fork() safely by making sure the threads are put in a safe state (no critical locks retained) when the fork() call happens. Also note that many things that should normally be unsafe, like having running threads calling malloc() while another thread is forking, are actually safe in the real world using certain implementation of fork, since there are pre/post fork "hooks" in the malloc implementation in order to fix the state of the child.

So if you control very closely the libs you link with and what they do, as well as the threads you use yourself, it is possible to use fork() in a reliable way.

<comment><USER12> glibc attempts this using the pthread_atfork() hook. it takes the global malloc (and possibly other) forks in the parent before forking, then both the parent and child release the lock before returning to the callee's code. obviously if you have locks in your own code then you may or may not need to hold these before calling fork() so that both parent and child can release them after the fork().

the apple way of just throwing its hands in the air and abort()ing sound either like giving up because the apple devs think it is too hard, or because they feel the need to impose their way in crippling code that uses fork() without calling exec()

in my opinion if you are going to write a library that uses threads and is supposed to be thread safe then remember you need to handle the issue of fork(); don't be lazy, do the job properly, even if doing it right it is hard and ugly.

</comment></comment><comment><USER13> The fork-exec code in my program currently does some things that are supposedly unsafe, but I can't figure what to use instead of initgroups() which is not async-signal-safe. I want to run the child as a different user for which I do initgroups+setgid+setuid between fork and exec. The only solution I see is to run getgrouplist() before the fork then in the child use setgroups() instead of initgroups(), but both of these functions are non- standard.

EDIT: Never mind, seems like initgroups() is also nonstandard but generally available on Unix-like systems.

</comment><comment><USER14> Hmm, consider we'd want to keep fork(), then the only safe way to deal with this would be, that critical sections were actually transactions implemented on the OS level and at after fork() in the child all transactions in flight get rolled back before returning from fork().

I see two implementation challenges with that suggestion:

1.) implementing that transaction mechanism as a kernel feature: When entering a CS mark all pages CoW, upon leaving the CS merge modified pages (problem: Whole pages are then mutual exclusive, dealing with this is the challenge)

2.) battling with user space implemented locks that use atomics.

\-----

An immediate mitigation I see is, that fork() itself is a CS on _all_ the locks of a process. If we consider that only the standard locking mechanisms are used, then whenever a CS is entered (which includes the creation process of a locking primitive) it raises/posts a global fork-lock semaphore. And upon leaving that semaphore is lowered.

This still leaves the DIY-locking primitives problem open. But it should be more or less straightforward to add this to the system libc/pthread libraries' locking primitive implementation and fork() syscall wrappers.

Or did I miss something essential here? Talk is cheap, so if nobody has any obvious objections I'd actually go ahead implement it.

EDIT: Okay, one immediate problem I see is, that this would pose a challenge for calling fork inside a CS. Technically this is a situation where thread recursive locks would help, but as we all know, recursive locks are highly problematic.

<comment><USER8> Sounds like you are trying to re-invent Software Transactional Memory - was a nice idea 10 years ago, but after lots of research it is still not available in common runtimes...

Some more problems to consider:

1\. how do you roll back IO that happens in a critical section? 2\. your global semaphore will be highly contended and basically destroy scalability

<comment><USER14> re 1) good point, didn't think of that

re 2) the fork global lock satisfies every requirement for a multiple- readers/single-writer, with fork() being the only writer. Unless the program does a lot of fork()-ing it should hardly ever run into contention. The moment fork() waits on the lock, further attempts on read-lock are delayed until after fork() completes and the existing reads are waited for, before fork()-ing.

</comment></comment></comment><comment><USER15> Endorsed.

Even fork _with_ exec can be real trouble. This is one of my bugaboos at work: due to poor life choices and high pain tolerance, I own the infrastructure we use to spawn subprocesses (carefully.) For various reasons (security most notably) we have to do some very tricky things in and to the forked child before exec(), and pretty much all of this code is a disaster waiting to happen. Every so often I get feature requests for more stupid pet tricks people would like out of subprocesses, and they're always surprised by what their "simple" change would entail.

I'd like it if Linux had native support for posix_spawn, but even that would require a lot of extensions to be useful.

Don't get me started on the teams that want to break forking rules and thus ask me how to guarantee a process has no non-main threads. There are few ways you can make me more upset than by building software that breaks if some one else happens to call pthread_create and doesn't tell you.

</comment><comment><USER16> I ran into this exact problem recently in a multithreaded Python app and spent two days trying to figure out wtf was going on. The multiprocessing spawn startup mode that was added in 3.4 solves this for most use cases at the expense of a small performance hit. For 2.x you are SOL however.

<comment><USER7> Part of my motivation for writing these things is because after I've wasted so much time, I'd love to help others not do the same. Hopefully the article shows up when you search for the right error message. Its also so _I_ remember what the heck was happening.

Re: multiprocessing in 3.4: I think it is unfortunate that due to backwards compatibility, the "forkserver" mode can't become the default on Unix, since it would help avoid this particular issue.

</comment></comment><comment><USER17> Face the fact, fork() is fundamentally flawed!

<comment><USER18> I would like to see the reasons this comment was down-voted. It is true that fork(2) is simple and elegant design, but it involves a huge accidental complexity and hidden implications like in the original article. There is a reason that Linux has clone(2) underneath.

<comment><USER19> > It is true that fork(2) is simple and elegant design

What exactly is simple and elegant about it? Have you looked into how much tooling is necessary everywhere in unix to make it work? It's insane. It has a huge footprint and it does not provide standardized APIs to make it work for non covered cases (the best we have is pthread_atfork which is not portable).

<comment><USER20> > What exactly is simple and elegant about it?

Spawn-style APIs have to take a billion parameters that are mostly set to defaults, for things like working directory, environment variables, user ID, controlling terminal. Whereas fork+exec means you can express these things in a more compositional, buildery style: fork, change the two things you actually need to change, then exec.

<comment><USER21> I don't think this makes the case for elegance at all.

Global vars set wherever in a program are preferable to explicit params when creating a new process? And there's no way to have default params or anything?

<comment><USER20> > Global vars set wherever in a program are preferable to explicit params when > creating a new process?

The outside world is always going to be implicit global mutable state; spawning an arbitrary process to do who-knows-what is inherently that kind of problem. If you could express it as an actual function you wouldn't need to run a separate process.

> And there's no way to have default params or anything?

Well not in C. You can sort-of do it by having a struct with fields that you fill in where all-bits-zero is the default but that's generally more trouble than it's worth.

<comment><USER21> Well by that reasoning there's never any need for function purity since you can't eliminate global state on current computer designs.

C easily allows you to have multiple functions. exec_with_current_state or something. And a struct is a way, too.

<comment><USER20> > Well by that reasoning there's never any need for function purity since you > can't eliminate global state on current computer designs.

If you care about purity you don't use spawn-like functionality at all.

> C easily allows you to have multiple functions. exec_with_current_state or > something.

That doesn't help. You'd need to have one variant where you just want to change the working directory, one variant where you just wanted to set an environment variable, one variant where you want to do both, one variant where you want to run as a different UID and change working directory but not touch the environment variables...

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER22> fork() is not per se "dangerous" in such context. You just have to be careful enough and only use asynchronous-signal-safe fonctions, as you would do in a signal handler (see [https://www.securecoding.cert.org/confluence/display/c/SIG30...](https://www.securecoding.cert.org/confluence/display/c/SIG30-C.+Call+only+asynchronous- safe+functions+within+signal+handlers))

Typically calls such as malloc(), printf() etc. are strictly forbidden in the child after a fork().

<comment><USER23> Only if your app is multi-threaded. Unfortunately, many apps are multi- threaded "under the covers" due to libraries that spawn threads, so you need to be careful.

This post does the best job explaining the issue: [http://www.linuxprogrammingblog.com/threads-and-fork- think-t...](http://www.linuxprogrammingblog.com/threads-and-fork-think-twice- before-using-them)

</comment></comment><comment><USER24> If there are specific issues with locks from dead child processes that deadlock, maybe the locks could be addressed specifically.

</comment><comment><USER25> > How to use fork safely: 1. Only use fork to immediately call exec. 2. Fork a > worker at the beginning of your program, before there can be other threads. > 3. Only use fork in toy programs.

4\. Stop writing broken multithreaded code altogether or if you must at least run an event loop per core/thread and use a wrapper for fork() to put the system into a fork()able state before forking. It's nice and reliable.

Multithreading by itself is just not a high-level concept to be used reliably by programs.

<comment><USER26> Fork/Exec is only good if you are going to run a _different_ program then you are currently executing.

If you want a series of worker threads that are _effectively_ the same as the master/initial thread. The proper way to do this clone(2) not fork, or fork/exec.

When used properly clone allows your group of threads to share a single PID, and TGID (Thread Group IDentifier). This cuts down on kernel resources your process(es) are using. They can natively share file descriptors and memory between each other (improving cache coherence). Also their signals are handled globally for all threads at once, not each thread managing it's own signals like Fork/Exec will result in.

<comment><USER9> ...And why would you want that? The advantage for fork(2) over clone is that you CAN'T share those datastructures, making it harder to write code resulting a deadlock: you have to share explicitely.

As for the kernel resources for the process, you'd be surprised how large RAM has gotten these days...

<comment><USER26>
       ...And why would you want that?

All your applications share the same virtual memory space. So you have a much higher hit rate in your TLB, then without. It greatly lowers the chance that shared memory will leave L3 cache.

Cache hit rates are very important...

         The advantage for fork(2) over clone is that you CAN'T
         share those datastructures, making it harder to write
         code resulting a deadlock

Dead locking has nothing to do with how RAM is accessed or how virtual memory is partitioned, it has to do with how you are managing your locking. Modern memory/instruction re-ordering is stupidly fast. If you are locking memory/data structure locations you are likely doing something wrong. Concurrent Memory fences are around 3 orders of magnitude faster then locks.

         As for the kernel resources for the process, you'd be
         surprised how large RAM has gotten these days...

TLB/L2/L3 cache is still premium real-estate. RAM size is awesome! I love waiting 100,000cycles to get the page I requested.

<comment><USER9> Cache hit rates aren't as important as you seem to think. If you're not google, and your app doesn't have hard/soft realtime constraints (videogames, flight control systems, etc.) your app will probably be fast enough. You'd be surprised how fast computers are nowadays...

      >If you are locking memory/data structure
      locations you are likely doing something wrong.

Oh, concurrent memory fences! Well, that solves all my problems. Let's see. All I have to do is give up all hope of my code ever being portable to mutiple architectures, and than dig around in asm to add support for them to my environment, which can range from mildly annoying (C), to near-impossible (JVM). Or I can use fork(2), and do what programmers have been doing for decades: trading performance for simplicity, and productivity.

That'll be a _really_ hard decision.

<comment><USER26>
        All I have to do is give up all hope of my code ever
        being portable to multiple architectures

You know nothing about modern compiler atomics.

Post 2011 compilers (LLVM, GCC, MSVC, ICC) standardized generic memory fences for C++ and C. These are fully portable as the compiler itself determines how the layout of Acquire/Releases needs to be changed on platform you are compiling too.

Atomics are supported on ARM, x64, MIPS64, SPARC64, POWER8, POWER9...

So they're very portable. The compiler even manages removing unneeded fences. Like if you add an acquire fence after a CAS load. The CAS load is an acquire fence on x64, but not on POWER8.

<comment><USER9> Okay, what about HLLs? Memory fences don't work if you aren't writing C. And as I've said, forks are an easier way to get these guarantees, albeit at a moderate perf cost.

<comment><USER26> Seeing as this a conversation thread about raw system calls I assume we were working C not a HLL.

Are you just a professional contrarian?

Why are you concerned about system calls in a HLL?

Why are you concerned with concurrency guarantees in a HLL?

You care about none of these, you care about your run time. This is why you are working in a HLL.

<comment><USER9>
      Are you just a professional contrarian?

No.

      Why are you concerned about system calls in a HLL?

Because in many HLLs, fork(2) is the best (or sometimes the _only_ ) option for concurrency. As in C, it is certainly the simplest.

Many things written in HLLs use fork. Unicorn is perhaps the most famous example.

      Why are you concerned with concurrency guarantees in a HLL?

Because if I have multiple threads of execution, I want them to run in parallel if possible, and I want to minimize the risk of deadlocks. fork(2) does both.

      You care about none of these, you care about your run time. This is why you are working in a HLL.

No. I care about them, as demonstrated above. And I don't know what you mean by care about my runtime. I would go so far as to claim that the above is an insult to me and everybody else who uses an HLL.

I care about getting stuff done. If C is the right tool for that, I'll use it. If an HLL is the right tool, I'll use it. And if syscalls are the right tool for the job, I'll use them, too.

</comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER27> [https://en.wikipedia.org/wiki/Thrashing_(computer_science)](https://en.wikipedia.org/wiki/Thrashing_\(computer_science\))</comment>


Roberta – Learning with Robots - <AUTHOR> http://www.open-roberta.org/en/welcome/

<comment><USER2> So far the process to get the EV3 running with Open Roberta take some while... but we are working hard on that.

Mid september will be a new release available and with that some really cool features are available...

Everyone is also invited to join our Google group: [https://groups.google.com/forum/#!forum/open- roberta](https://groups.google.com/forum/#!forum/open-roberta)

To get a deeper look "insight" Roberta follow these links: [http://dev.open- roberta.org](http://dev.open-roberta.org) [http://wiki.open- roberta.org](http://wiki.open-roberta.org)

Enjoy Thor

</comment><comment><USER2> great thx!

One note: The link to the programming envirnment is: [http://lab.open- roberta.org](http://lab.open-roberta.org)

/thor

</comment><comment><USER3> There's also [http://www.robotmesh.com/project](http://www.robotmesh.com/project) which is Blockly and Python for VEX robotics

<comment><AUTHOR> Roberta is 100% free software, though.

You can also install the server component in your school, your lab whatever.

</comment></comment><comment><AUTHOR> More robots should be supported in the future. A Python version for EV3 is in the making.</comment>


Photos Capture the World's Sewer Systems When They Were Brand New - <AUTHOR> https://www.smithsonianmag.com/innovation/these-photos-capture-worlds-sewer-systems-when-they-were-brand-new-180973921/ <comment><USER2> And no photos.

<comment><USER3> there is a carousel of photos at the top.

what I want to see is what they look like now!</comment></comment>


Show HN: Writing Streak – write fiction every day - <AUTHOR> http://writingstreak.io/?hn <comment><AUTHOR> Hey, everyone! Super excited to make it to the front page of HN =)

I want writingstreak.io to help people to practice their writing skills and develop a habit of writing regularly.

It keeps track of daily word count, gives you writing prompts, has a github- like calendar representing the amount of words you have written each day, and all sorts of cool features to help you focus on writing.

I think it's gonna be pretty great.

It is in the very early stage at this point, and I would really love some feedback.

Does the idea make sense, do you think it's useful? What other features would you like in that kind of tool?

</comment><comment><USER2> Is this something only for people who want to _start_ writing, and need to start building the habit to put in time every day? Or can it also be useful for people who already write?

The main issue that pops out to me is that people who write have projects (sometimes several, sometimes massive ones), and "writing" daily sometimes means much more editing/cutting than writing fresh text (I'm married to a novelist; her first book lost about five hundred pages between the first "completed" draft and the final result). Or -- writing may mean composing new text, but it's placed somewhere in a much larger work.

Last thought! Be very careful about having ability to export from the start, and be very sure you don't run out of time / hosting fees / whatever and let it die with anyone's work trapped in it.

<comment><AUTHOR> If you are a professional writer you still might use some additional motivation or habit tracking. I have not yet put much attention into tracking number of words on bigger projects, but I definitely will.

If you don't need that sort of stuff and just want to publish fiction - I am also working on a related project - [http://fictionhub.io](http://fictionhub.io)

It is using the same platform, it's goal is to be a great place to publish high quality fiction(and, in the future, sell it as well).

And yes, I'll definitely be very careful with user data. Adding the export feature is on my todo list.

</comment></comment><comment><USER3> Mobile? Context: my daughter (12) wrote her first 5000+ word story on her Android phone without anyone prompting her to do it. But perhaps teenagers and younger are not your target audience?

<comment><AUTHOR> Wow, that's really awesome!

Regarding mobile - right now I want to focus on improving the website and making sure that it has everything users want and is really convenient to use. Currently it does work on mobile(but not throughly tested yet), I'll try to make it work really well.

At the moment I don't have any plans for an app, but if users will really need it I can make it.

</comment></comment><comment><USER4> First of all congratulations on launching. I like the idea and do hope one day to write my great American novel. In my past life I fancied myself a screenwriter (wrote six and produced one).

My three comments are:

* A screenwriting feature so that it counts pages over words.

* If you do support screenwriting then support the Fountain format. (I have a screenwriting/production app and I modified a WordPress plugin for that support.)

* NaNoWriMo is a time to shine. Start gearing up an awareness campaign in September.

<comment><AUTHOR> Thanks for the feedback!

Yeah, I have been thinking about adding screenwriting support. For now it makes more sense to just focus on fiction, but it's very possible that I'll add the Fountain support in the future.

I'll think what I can do for NaNoWriMo. Now I'm focused on short stories, but it would be really awesome to come up with tools that would help writers to work on bigger projects. I'll just need to figure out a simple and elegant way to do that.

</comment></comment><comment><USER5> I think this is kinda neat, but I think it's interesting because the GitHub contribution graph, which I see a similar graph to here, has been derided by a large group of people: [https://github.com/isaacs/github/issues/627](https://github.com/isaacs/github/issues/627)

I am actually a big fan of this "just do it" method of getting better or more comfortable with something though, and I think this is cool.

</comment><comment><USER6> I think this makes sense for a lot of people here, and that people will be able to sell books on Kindle store writing like this, but it is surely sad that writing has become this nowadays.

<comment><AUTHOR> Can you elaborate? Why do you think that?

(Maybe I didn't make it clear - the purpose here is not to "sell kindle books", but to practice writing skills. Not to generate as much content as possible, but to get better at your craft.)

<comment><USER7> Well put. Wríting for me is more of a trade than an art. You need to work on it to really get good at it. I'll give it a try, thanks.

</comment></comment></comment><comment><USER8> > You begin by writng a paragraph describing a settng of your story, and > continue from there:

> The most actve writers get a place on our leaderboard of awesomeness:

Looks like you may have a sticky 'i' key.

<comment><AUTHOR> Oh, thanks a lot, fixed!

<comment><USER2> A few other minor typos I noticed (since this seems to be the typo thread!):

for today(the green circle) (missing space)

On you profile page (your)

<comment><AUTHOR> Thank you very much! Corrected.

</comment></comment></comment></comment><comment><USER9> The sprint feature reminds me of this: [http://www.themostdangerouswritingapp.com/](http://www.themostdangerouswritingapp.com/)

Discussion on ^ link: [https://news.ycombinator.com/item?id=11197190](https://news.ycombinator.com/item?id=11197190)

Like the comments linked, I don't like how it just deletes all the work -- what if you'd written something really compelling and lost it all?

<comment><AUTHOR> Well, that's the point of the game) Otherwise when you stop writing nothing happens, no stakes.

Of course you can just ignore this feature and simply write without it.

I have also experimented with using a "pomodoro" feature instead of this(just a 20 min countdown timer), but sprints seem more fun. Maybe I'll add a preference allowing people to choose pomodoro instead.

</comment></comment><comment><USER10> This looks pretty cool. I think it would be fantastic if you'd integrate social logins here. I'd rather log in with Facebook or Twitter than create a completely separate account just to use the site.

Having a demo of sorts would also be useful, so that I could get a feel for the sprint editor. Animations are great, but it doesn't give me a sense of how well it would work for me.

</comment><comment><USER11> See also:

[https://750words.com/](https://750words.com/) similar concept, encourages you to write ~3 pages every day.

[http://nanowrimo.org/](http://nanowrimo.org/) national novel writing month.

</comment><comment><USER12> Could I set my own subject? It looks like a nice UX to help with general writing too.

<comment><AUTHOR> Yes, sure! Prompts are there to help people to get started, but feel free to ignore them if you want.

In the future I will probably add preferences allowing people to disable the functionality they don't use.

<comment><USER12> Just gave it a try and its fun. I had not realized this is a sort of community type of place. Anyhow, I like the goal driven GUI. It is really motivating and works. The title input is buggy in Firefox/OSX (text doesn't show up), and the main textarea does not scroll down as I type more than what the size will let. But those are silly bugs. This is fun. In fact, I think this could be turned into a product if you can package it as a stand alone application (not mobile app) for desktops. Leave out the public community aspect but keep the goals GUI.

</comment></comment></comment><comment><USER13> Nice. Now if only we could get GRRM or Patrick Rothfuss to use this tool.

</comment><comment><USER14> The problem I could foresee with this is that all/most writing prompts are just really irritating. A photo or something would be preferable.

<comment><AUTHOR> Thank you for the feedback! I am still thinking about the right format for prompts, I will definitely try to improve them.

Maybe I will allow users to submit their own prompts, or just have a randomized setting, character, and problem/goal.

Image prompts are an interesting idea, they never worked for me personally, but I'll think about adding them as an option.

Currently many prompts are taken from the top posts on /r/WritingPrompts, I'm an active user of that subreddit and I find them really helpful(they are usually specific and funny).

Do you have any advice on making a prompt good? Is there a kind of text prompts that you like?

</comment></comment><comment><USER9> The writing prompts are rather specific; I think a good writing prompt should be more open to interpretation.</comment>


Show HN: Product Hunt – download a list of users who upvoted a product - <AUTHOR> https://product-hunt-upvotes.herokuapp.com/ <comment><AUTHOR> get the source here: [https://github.com/ryanckulp/product_hunt_upvotes](https://github.com/ryanckulp/product_hunt_upvotes)</comment>


Ask YC: Why aren't high-rises built by machines? - <AUTHOR>

I have zero background in construction, but was just thinking about this...
It seems like erecting a high-rise is pretty well-defined by architects and engineers-- so why can't machines be built to automate much of the process? Even if that meant having a very  standardized building, it seems like it would be so cost efficient that the aesthetic compromise would be alright.
I know there is pre-fab housing.. it's cheaper, but the savings are negligible. But it seems like economy of scale would really kick in with large buildings. I know it would be expensive initially. Plenty of new R&D and engineering hacks.
I just want a giant machine that I put on a plot of land, feed it concrete and glass like an easy-bake oven, and 3 weeks later I have a giant high rise.
Why hasn't this happened? <comment><USER2> _Even if that meant having a very standardized building, it seems like it would be so cost efficient that the aesthetic compromise would be alright._

Skyscraper tend to be designed to be iconic, e.g. non-standardized. Building codes evolve, which also defeats standardized construction.

The new Freedom Tower (One of the World Trade Center replacements; a symbolic 1776 feet high), a hyper-iconic building, is lately estimated to cost around $1,000/sqft of floor area to construct. Simpler skyscrapers can cost as little as $200/sqft, so that's your competition. Your proposed venture needs to be able to 1) complete skyscraper projects at substantially less than $200/sqft, 2) meet building codes, and 3) convince authorities to grant construction permits for a radical construction technology.

Skyscraper construction, like house construction, is already automated to a great degree. Parts are mass produced in centralized factories, shipped to the construction sites, and put together like parts from a kit. Here is the brand new Trump Chicago being built with a helicopter delivering prefabricated parts: [http://forum.skyscraperpage.com/showthread.php?s=000ea7e2e81...](http://forum.skyscraperpage.com/showthread.php?s=000ea7e2e819b916c360907870a05ae9&t=128533&page=296)

</comment><comment><USER3> I would suggest the major problem with constructing buildings like that automatically is that each floor has to be carefully prepared with reinforcing material long before the concrete is poured. I think for this reason alone you're going to find trouble.

While the arrangement of these rods is a fairly straightforward but tedious process for a human to do, it's particularly tricky for a machine to manage because of the highly variable nature of the structure.

If you had some kind of construction material that didn't require internal reinforcing, where it could be just applied as-is, then you would see a lot more automation.

As others have said, building codes and quality control would have to play a part here. A lot of automated manufacturing is subject to strict checks, but usually the resolution for a defect is to discard the part. In the case of a building, or even a part of it, that may not be so easily accomplished.

</comment><comment><USER2> _why can't machines be built to automate much of the process [of erecting a high-rise]?_

This might be a good place to look for answers: <http://forum.skyscraperpage.com>

The engineering section discusses construction technology.

<comment><AUTHOR> That's spot on, really interesting discussions.

</comment></comment><comment><USER4> There have been a few recent developments (I remember hearing about a concrete 3D printer), but it seems that the technology just isn't there yet. The problem is that the robots need to be small (enough to be transportable and reusable), cheap (cheaper than construction workers) and mobile (small => you need many to slowly build up the building). Most robots don't have that level of sophistication yet. All the manufacturing robots are stationary and (relatively) large, and mobility is something of an unsolved problem.

This is not to say that it is impossible, just that I think there are significant engineering hurdles to robots building skyscrapers (maybe smaller houses first?).

</comment><comment><USER5> It makes most sense to manufacture things when they're small, cheap, and made in large quantities. High-rises are at the other extreme.

</comment><comment><USER6> I saw 0.01 version of one in San Mateo, CA about 20 years ago. The building was built from the top down. Each floor was built on the ground, cast concrete, I believe, and hoisted into place underneath the one on top. It probably didnt have any real advantages, because I havent seen it done again.

One would thing that simple, but well designed and constructed, housing would dominate, but it hasnt happened. Construction codes are one thing, but I think the real reason is that nothing can compete with fine contros and flexibility of humans when it comes to building large structures.

</comment><comment><USER7> There attempting to build something like this, [http://www.engadget.com/2007/01/15/rival-robots-prepping- to-...](http://www.engadget.com/2007/01/15/rival-robots-prepping-to-automate- home-building/) . Personally I think its a horrible idea, I have a gross aversion to suburbia clones, let alone soviet style skyscrapers that all look the same. But I suppose it would be interesting. As gravitycop pointed out skyscrapers more often than not are ego plays so the more outlandish the better.

</comment><comment><USER8> the machine does not have to be big... it can be a bunch of small spider like machines, or even more groups of machines: \- material suppliers \- welders \- concrete ones (these can be essentially carrying tubes that are pump out concrete)

this might be the next step for building multilevel farms, for agriculture (building huge pyramids). I've seen mockups/studies done in Japan (I think) that talk about exactly this, spider like swarms of robots moving along rail like structures maintaining/buidling fractal like pyramid structures that are super-tall ;))

the only way to take agriculture/housing to the next level... have multi-level cities (and not what pseudo-multilevel that we have now)... \- rail constructors (rails for welders/material suppliers to move along)

<comment><AUTHOR> Spiders are a fascinating if not slightly scary concept. I would if current construction methods would have to be simplified in order to reduce errors.

</comment></comment><comment><USER9> maybe you are underestimating the extent to which this is already done? there are machines that eat steel and glass and produce windows and structural elements (wires, carpets, tiles, lightbulbs...)

the economies of scale come from not lugging these machines to every job site.

</comment><comment><USER10> I think that the cost of human labor is <10% of the cost of materials, so it would not make much of a difference.

I wish I could find some numbers for some recent projects, I haven't got anything to back this up.</comment>


Anyone in the UK using the Long Week to start a startup? - <AUTHOR> http://www.barricane.com/2011/03/05/uk-internet-startup-week.html

<comment><USER2> I like the idea! Some of those days I've already got other stuff booked in, but luckily I've already started working on The Next Big Thing ( _cough_ ). Those days are perfect though to get a lot of stuff done.

</comment><comment><USER3> I'm practically unemployed and don't give 2 hoot about long weekends or royal weddings - so I'm in :)

Great idea.

</comment><comment><USER4> I'm in but I'm missing the "great idea" bit

</comment><comment><USER5> i am in! mailing you right away</comment>


Resources for your Business Plan and Presentation - <AUTHOR> http://academicvc.blogspot.com/2008/02/raising-capital-part-08.html

<comment><USER2> Ignore this. The best thing you can do to attract VCs and raise capital is launch a service and begin engaging your users in the marketplace. There is no substitute for real traction.</comment>


Ask HN: Is Hacker News' rss feed broken - <AUTHOR>

I'm using feedly with https://news.ycombinator.com/rss as the source and the last article I have in the feed is https://news.ycombinator.com/item?id=6639989. Other feeds are working fine. Is anyone else experiencing any problems with HN's feed? <comment><USER2> I've been staring at my Feedly since today morning, waiting for Hacker News to update.

Oh well...Guess I might as well get some work done while I still can...

</comment><comment><USER3> Same here, i've noticed feedly hasn't been playing well with hacker news for the last couple of days.

</comment><comment><USER4> I use protopage.com and the same source. It's been staying up to date for me.</comment>


Place Cage - Nicholas Cage Placeholder Images - <AUTHOR> http://www.placecage.com/

<comment><USER2> Awesome. I use Bill Murray for all my placeholders. Gonna start using this, but Place Bill would make my day.

<comment><USER3> That's a great idea. Still need to find more images, but here you go: <http://www.fillmurray.com/></comment></comment>


CloudApp Now Restricts Free Plans to 10 Drops/Month - <AUTHOR> https://www.getcloudapp.com/plans/ <comment><USER2> Get Minbox instead! Free screenshots, file uploads (use CMD+U to upload any file from finder), and more. [http://minbox.com](http://minbox.com)</comment>


Glenn Greenwald on Livestream - <AUTHOR> http://new.livestream.com/accounts/4448875/GlennGreenwaldSpeaksOut

<comment><USER2> Watched for 5 min, Greenwald is still butthurt over David Gregory

<comment><USER3> You don't have to question a reporters right to report just like you don't have to question global warming, or equal rights for minorities or that the earth is round. Sometimes there are dumb questions and David Gregory obviously asked one.</comment></comment>


Pilot's Spilt Coffee Helps Trigger False Terrorist  - <AUTHOR> http://spectrum.ieee.org/riskfactor/aerospace/aviation/fate-is-almost-the-hunter-united-airlines-pilots-spilt-coffee-helps-trigger-false-terrorist-alert-

<comment><AUTHOR> I'm not sure the claim (title) is entirely accurate since human error definitely plays a part in the problem.

The mentioned WSJ article is here:

[http://online.wsj.com/article/SB1000142405274870367590457606...](http://online.wsj.com/article/SB10001424052748703675904576064112406869284.html)</comment>