Stanford CS Book: Mining of Massive Datasets [pdf] - <AUTHOR> http://infolab.stanford.edu/~ullman/pub/book.pdf

<comment><USER2> Pretty cool. I work in search quality at Google, and this is a pretty decent overview of the more universal tricks I've picked up from people on the job, as well as a lot of things I didn't know. MinHashing in particular is one of my favorites.

Also, if you like this, I'm trying to collect resources of this quality in this subreddit: <http://www.reddit.com/r/learnit>

<comment><USER3> Thanks for sharing! This is a great set of resources.

</comment></comment><comment><USER4> If you guys mostly have CS background you should definitely check this out : <http://www-stat.stanford.edu/~tibs/ElemStatLearn/>

It's Data Mining from a statistician point of view. You can download the entire book for free on the website and the graphs and computations are done with R.

This book is also one of Hal Varian's favourites: <http://www.dataspora.com/blog/sexy-data-geeks/>

</comment><comment><USER5> I've been thinking lately of (finally) pursuing graduate studies, and data mining is an area that I find drawn to. Obviously Stanford is doing some significant research in this area, but I've been out of academia for 4 years and I somehow doubt I'd be a competitive applicant. Does anybody have personal experience with other universities/programs that are doing extensive research that they'd like to share? It'd be greatly appreciated!

<comment><USER6> I was out of school for 3 years before going back to pursue a PhD. It can be done, but you're going to have to put in a _lot_ of work the first year getting back into the swing of academia, and even more if you're going back to specialize in something you don't already do.

</comment></comment><comment><USER7> Anyone have a epub or Mobi format? I just got a kindle and the converter for it ruins code samples if you take it from PDF.

<comment><USER8> I used to face the same issue with my Kindle and bought an iPad, because of how frustrating I found this, and am quite happy with the Kindle app on it.

Very smart of Amazon to hedge themselves on the hardware so cleanly.

Sharing my story not to diss the device, but in case anyone else is having the same predicament and considering jumping.

<comment><USER2> Whenever I asked around on academic circles about this, the ipad was the consensus. The kindle DX does an ok job according to the people I talked to, but the slow page turning drove people crazy when trying to understand a complicated paper referencing previous math/diagrams.

I really wish the academic world wasn't standardized around formats that only work for print. There are some LaTeX->html converters out there, that could presumably be used to make epub, but I have no idea how well they work.

<comment><USER9> Your peers are generally correct. I used a DX for technical papers and PDFs after going through the conversion hell you are stuck in (albeit with a Sony Reader) and then jumped to an iPad when it came out. The best advice I can offer is to hop over to mobileread.com and check out the forums to see what the current state of the art is. Back when I was doing two or three conversions a day the tools to use were Rastafarian and PDF2LRF but I am guessing there are better options available now. You may also want check out Calibre, which is an ebook library management app but one with a lot of built- in conversion routines.

<comment><USER10> Holy cow, could a project have picked a worse name than "Rastafarian"? If one searches for "Rastafarian", you can guess what turns up. But even "Rastafarian pdf conversion" seems to drop the "pdf" and returns lots of results for converting to the Rastafarian religion. Same for "convert".

I guess I'll have to go trolling through the mobileread.com forums, but just damn.

</comment></comment></comment></comment></comment><comment><USER11> Chapter 2 of this book is awesome. Covers specific implementation and design strategies for map reduce matrix multiplication and table joins.

</comment><comment><USER12> Thanks, great idea for the reddit group.

</comment><comment><USER13> An amazing resource. Brilliant... Stanford lead the world in open education (MIT are great too)</comment>


Introducing Trails (twilio-on-rails) - <AUTHOR> http://blog.twilio.com/2010/02/introducing-trails-twilioonrails.html

<comment><USER2> Seems like a good way to leverage the platform!

I'm still waiting for Twilio to accept payments outside the US though... Only limited to the trial otherwise. Hopefully it'll happen someday!

<comment><USER3> Twilio took my money quite happily, and I'm in Canada. Probably depends on who provides their merchant stack.

</comment></comment><comment><USER4> really great tool. Hope to use this soon for one of my applications!</comment>


My week-long fast (2013) - <AUTHOR> http://aeon.co/magazine/health/s-abbas-raza-fasting/ <comment><USER2> Interesting article but what really grinds my gears is the following:

>My wife and I don't normally eat breakfast (she has a cup of coffee and I drink a Coke Zero — yes, yes, I know it's bad)

If you go through such effort to do a heavy crash diet and your base diet is fundamentally flawed why do you not fix your base diet first? Getting a proper meal cycle in there for the rest of your life is far more beneficial than reaping the benefits of a crash diet.

This especially bothers me when I hear people say "but I cannot loose weight!". Look at your base diet. Just freaking write on a piece of paper what you eat after you eat it for a week and you will be a lot more conciousness about it.

</comment><comment><USER3> Complains about the science behind existing fasting regimes, then combines his fast with additional exercise.

FWIW, 5:2 has worked for me. 10KG in 9 weeks with only positive side effects to date. It feels sustainable. Would recommend.

<comment><USER4> Interesting. How long have you been doing it?

<comment><USER3> Just the 9 weeks so far.</comment></comment></comment>


Ask HN: Why doesn't HN have notifications? - <AUTHOR> I love the conversation and debate that goes on here but I don't always go back to check if people responded to my comments. Wouldn't comment notifications be a really helpful feature of HN? <comment><USER2> Yes, but HN is a zero-distraction distraction: It's a workplace procrastination tool that doesn't let the user veer too far from their work. You can see this in the anti-procrast settings and the lack of any advertising here (both of which are very nice features I use regularly). If HN had notifications it'd lead on a slippery slope to HN being a social network instead of a news aggregator, and I don't think anybody wants that.

</comment><comment><USER3> This exists already, make sure to google it. [http://www.hnreplies.com/](http://www.hnreplies.com/)

<comment><USER4> That's a third-party site though. If HN did a very simple thing: publish some data in RSS feed format, then an RSS reader (e.g. one living in your browser as an extension) would provide a notification interface.

</comment></comment><comment><USER5> Build your own:

[https://github.com/HackerNews/API](https://github.com/HackerNews/API)

<comment><USER6> I would prefer it if the API exposed the messages in the input format rather than HTML format.</comment></comment>


2013 Lewis meditation results - <AUTHOR> http://www.gwern.net/Lewis%20meditation <comment><USER2> It is really interesting to see the "Western" approach to meditation.

I grew up in a Buddhist community and have sat week long meditation retreats without anyone quantifying results.

It is interesting that Westerners have to be scientifically convinced that meditation will have a quantifiable outcome for it to be a helpful practice. It's like we know that eating fresh vegetables is good for us, but only when it is proven to us are we convinced that we should go ahead and eat fresh.

<comment><USER3> >> It's like we know that eating fresh vegetables is good for us, but only when it is proven to us are we convinced that we should go ahead and eat fresh.

How exactly do you think we came to know that eating fresh vegetables is good for us? The answer is science.

I mean, sure, humans have preferred fresh vegetables over non-fresh ones for ages, but that's because fresh looks, smells and tastes better than non-fresh.

<comment><USER4> 1) Can it be wiser often to rely on 'fresh looks' instead of science in our choices? Using scientific findings to make the right choice requires access to these findings, ability to interpret them properly, and certain time investment. Most people can't do that, especially considering that we face a lot of choices.

2) With practices such as meditation, there are no 'fresh looks' to judge it by, and scientific knowledge on the matter related to our mind and consciousness seems to be limited. How do I rationally decide whether or not should I meditate?

</comment></comment></comment><comment><USER5> I'm very excited at HN's discovery of gwern.

<comment><USER6> What exactly is gwern?

<comment><USER7> gwern is a guy! You can read about him here [http://www.gwern.net/Links](http://www.gwern.net/Links)

</comment></comment></comment><comment><USER8> I guess it won't be long until the Discovery Channel hosts "Meditating With The Stars" and "Ultimate Meditation Championship".

</comment><comment><USER9> Interesting to see such an amount of work done but it fails to take into account that a single month of meditation is not nearly enough to even begin to properly feel the beneficial effects it can have.

In some areas, simply feeling might also still be a better choice than quantifying.

</comment><comment><USER10> Interesting analysis but I think the last sentence sums up my sentiments: "Even if all the other issues could be dealt with and mindfulness meditation did improve arithmetic, what is it a valid proxy for?"

<comment><USER11> Improved arithmetic is a side effect. I doubt there is any way to meaningfully quantify the beneficial effects of mindfulness meditation. Any long-term practitioner would be able to tell you that.

<comment><USER12> Hmm. Tell Alan Wallace, the Shamatha Project, and the UC Davis Center for Mind and Brain! There's a lot of interesting variables and correlations to study in relation to meditation.

My Zen teacher talked about some Shamatha Project study that found sustained concentration meditation to decrease neuroticism and increase conscientiousness. Anecdotally, he confirmed that this is what he has seen as a meditation teacher.

I find it really weird to measure the efficacy of meditation by arithmetic puzzles, though. Is anyone claiming meditation makes you better at math? That doesn't seem like a big area of interest to me.</comment></comment></comment>


Drowning Doesn't Look Like Drowning (2013) - <AUTHOR> http://www.slate.com/articles/health_and_science/family/2013/06/rescuing_drowning_children_how_to_know_when_someone_is_in_trouble_in_the.html <comment><USER2> for a visual experience of this, check out [http://spotthedrowningchild.com/](http://spotthedrowningchild.com/)

note, this is a (I hope/assume) re-enacted video of a child drowning so maybe one to avoid if you've had related traumatic experiences.</comment>


The Problems of Open Source - <AUTHOR> http://www.lambdassociates.org/blog/the_problems_of_open_source.htm

<comment><USER2> the page has simply moved, that's why the dupe detector didn't catch it:

<http://news.ycombinator.com/item?id=736578>

<comment><AUTHOR> Interesting. Thanks</comment></comment>


A Collection of PostgreSQL Indexing Tips - <AUTHOR> http://www.craigkerstiens.com/2013/05/30/a-collection-of-indexing-tips/ <comment><USER2> Indexing _all_ columns and then reading the statistics about which indexes are being used is an interesting approach, but it is only able to give you simple single-column indexes. Postgres has multi-column (composite) indexes and can be very clever about using them; additionally, there are the functional and partial indexes.

A better approach, I think, would be a tool that takes an SQL log and figures out the optimal indexes based on those SQL statements. It could run "explain" statements on the recorded queries to determine which columns need indexes, including ones that are queried together, which would benefit from multi- column indexes. The "explain" output would also give some information about common filter values which could be used to generate partial indexes.

Another way would be to use genetic algorithms to create random indexes, run the queries on a dummy database, and then read the index usage stats to determine the most efficient combinations.

What would be lovely is a tool that you could run autonomously every week or month, and which would simply add/remove indexes as it sees fit.

The ultimate solution would be for Postgres itself to figure out and report potential plan improvements. After all, the planner knows best.

<comment><USER3> Original author here, the adding an index on every column is very much a joke. A better solution would be taking advantage of pg_stat_statements in Postgres 9.2. It normalizes all queries which are run and tracks total time spent in your DB on those queries, this would be a clear way to determine where indexes can likely improve your performance. There's actually an earlier post I did which talks a bit more about this - [http://www.craigkerstiens.com/2013/01/10/more-on-postgres- pe...](http://www.craigkerstiens.com/2013/01/10/more-on-postgres-performance/)

<comment><USER4> Features like pg_stat_statements are a major reason why I love Postgres. The dev team makes sure the foundations are rock solid, and then start pouring on the features. If only $work used Postgres...

</comment></comment></comment><comment><USER5> There's a few other features of Postgres indexing that are interesting:

\- Indexes over expressions (e.g. upper(value)) are very helpful.

\- You can add a WHERE clause to an index to index only part of the table, which is fantastically useful.

\- on a non-C locale, LIKE won't (ever) use indexes. varchar_pattern_ops can help you with this to an extent, but it's not perfect.

<comment><USER6> I think LIKE can use indexes where the left is anchored with enough text for an index to be useful. LIKE 'this is fun%' will be more likely than LIKE 'a%' LIKE '%fun vacation' won't but if you have an index on reverse(mycolumn) and you have where reverse(mycolumn) like reverse('%fun vacation'), it might.

You can also add pg_trgm as an extension and then create a GIN index. Not perfect at all but better than nothing.

Also indexes over expressions can be used to precalculate interesting sets if you have expensive routines in the database (the typical hypothetical example is image recognition).

<comment><USER5> > I think LIKE can use indexes where the left is anchored with enough text for > an index to be useful. LIKE 'this is fun%' will be more likely than LIKE > 'a%' LIKE '%fun vacation' won't but if you have an index on > reverse(mycolumn) and you have where reverse(mycolumn) like reverse('%fun > vacation'), it might.

That's the case for the C locale, but (as I understand it) not for other locales.

<comment><USER6> Reading a bit more it looks like you have to specify the text_pattern_ops on the index. This makes some sense actually, since it means that the index is ordered by the localized string order (so the index can be used).

</comment></comment></comment></comment><comment><USER7> To the article's author: I think you could go much deeper, and get into the discussion of what can be indexed and what can't, eg timestamp with or without timezone, and how indexability of a column or a function affect the design of the schema itself.

One solution I found after many struggles was to require all event tables to have a timestamp with timezone column, index its date() and require all queries attacking these tables to be selective on the date.

<comment><USER3> There's definitely a lot deeper I can go on this, this is largely a test of a series which could go quite deep. My initial post yesterday simply highlighted one method of indexing and out of that came many of the questions in here. In general Postgres is pretty flexible with what you can index with not many limitations, its impact on the rest of the schema aren't always so simple so would love to hear what you have in mind as impact. Please feel free to drop me a line and would love to turn more of that into a deeper post.

</comment></comment><comment><USER8> When you get into page splits and such I would think the I/O required to support indexes in transactions would be the most important cost. Has Postgres found a way around that?

Edit: I ask because it's not included in the article's "Costs of Indexing".</comment>


'horrendous failure' led founders to revolutionary... nearly broke. Repeatedly. - <AUTHOR> http://money.cnn.com/2009/09/03/smallbusiness/harmonix_rock_band_startup_story/

<comment><USER2> Just from reading that article it seems Alex and Eran, as founders, seem to really complement each other well. And their persistence, wow!

 _The company was on the rocks. We had zero revenue. We had been trying for four years to make something work. We were out of ideas. Those first four years had been a graveyard of misstarts and product concepts that never made it anywhere._

</comment><comment><USER3> Good article. Maybe you should edit the title to get more reads, though.</comment>


Rejection Therapy - The Real Life Game - <AUTHOR> http://rejectiontherapy.com

<comment><USER2> God damn such a good idea. Let's build a list of rejection ideas and maybe avoid paying $16. Here are mine.

* Ask a random person for their phone number without introducing yourself, explaining why, or anything.

* Ask someone to give you their seat in some public area (e.g. bus, restaurant). (Note: what if they actually give it to you? o_o)

* Be a beggar. Ask people for money.

* Ask someone of your preferred gender out.

* Ask someone not of your preferred gender out.

* Ask some store employee if it's alright for you to explore some employee-only area of the store, e.g. their back room, or behind a door marked "authorized personnel only".

* In general, go somewhere you're not supposed to be where someone is likely to discover you and kick you out.

* Go in to a store and ask to buy something they clearly don't sell (e.g. try to buy a slice of pizza from the cashier in a computer store).

* When you're leaving Point A for Point B, ask a stranger at Point A out of the blue if they'd like to travel to Point B with you.

* Go to an area where live music is being played. Ask the musicians to turn down their volume.

* Ask a store employee if they can turn off the music in the store while you're in the store.

* Apply for some sort of grant or scholarship that you are clearly not going to get.

* Mail a famous person a copy of a book they wrote with a request that they sign it and mail it back to you.

* Email the support team on a site like Ebay or Yahoo Answers asking for more karma. (NOT HN; I don't want PG to hate me (and you probably don't either!))

* Email me ([my username] at gmail dot com) asking for anything and I will reject you. (Unless you are feeling clever and ask not to be informed of my credit card number or something like that.)

* Ask people to upvote you on HN

* Volunteer for some organization like Greenpeace that collects donations in the streets. "Do you have a minute to talk about [insert issue]?" Save some time by inventing your own issue.

* Go to a massage place and ask for a free massage.

* Contact the maintainer of an open-source project you use and personally request that some feature be implemented.

* Ask someone on the street if you can have your picture taken with them.

* Make a "free hugs" sign and get rejected by all the people who ignore your sign.

* This is a bit off the wall, but I actually saw a guy doing this. Get a couple empty water bottles, put them up on your nipples and ask people to "milk" you by squeezing the water bottles.

* Sell something like paper cranes on the street. Say "thank you" to all the people who hear your pitch and don't buy a crane (like a creep).

* If you don't know how to fold a paper crane, stand on the street asking passerby if they can teach you how to fold them. (It'd be pretty funny to see someone asking for lessons on how to fold paper cranes one day and selling them on the next day.)

* Contact someone prominent, such as a professor who teaches a subject you're learning about. Ask them if you can buy them lunch (e.g. to pick the professor's brain about said subject).

* Come to some sort of event. Ask if you can give a quick presentation on something before the main event.

* Knock on random doors and say "please insult me". Pretty sure receiving insults is a form of rejection.

* Ask if you can bring a trojan horse inside an embassy. <http://www.youtube.com/watch?v=Xs3SfNANtig>

* Ask for directions to somewhere insanely far away--e.g. if you're in NY, ask how you can get from where you're standing to Tokyo.

I think it's important to separate the phase of thinking up rejection ideas from the phase of actually executing them. If you think about actually executing the ideas as you're thinking them up, your brain might sort of freeze up and refuse to produce more.

Please upvote this comment because it could potentially save a lot of people $16--in other words, perhaps 1 in 10 people who read this thread would have been willing to pay $16 to read it (and that's not even counting all the consumer surplus generated for those who value the deck of rejection ideas at less than $16). How many other HN comments would a significant number of readers value at $16?

<comment><USER3> Though the point and goal is obviously to get rejected, I prefer the rejection ideas that have a chance of nice outcomes if not rejected. So, the mail a famous person a book idea over asking for a seat.

Some additional ideas:

* Ask for a raise

* Ask your employer to send you to a conference

* Ask the conference holder for free admission to the conference

* Ask for a neck massage (even better if you don't know or just met the person)

* Offer to buy lunch for somebody you admire but have never met

* Ask a friend to do your laundry

* Create a reddit post saying it won't make it to the front page :)

* Leave your wallet at home one day. Ask starbucks for a free coffee, ask a coworker to loan you the cash for lunch, etc.

* Got any debt? Call your debtors and ask for it to be lowered.

Update: Looks like the parent comment has been updating their suggestions as I was typing, so there's some unintentional dups here.

</comment></comment><comment><USER4> > This may be true, since neither death or rejection are bad. They are portals > to other places.

Would it be wrong to take the meta-contrarian position on this one? Rejection, like death, kind of sucks.

<comment><AUTHOR> You're right. Rejection is hard, and people are terrified by it (why do you think cosmetic surgery is so prevalent?)

And yet, rejection usually consists of a simple and benign "no".

Why are we so sensitive to the word no? Can this be healthy, that we would hide in the shadows all our lives than ask for what we want and need?

The amazing thing about Rejection Therapy is that people are more likely to give you what you ask for than you can imagine. The world is yours for the taking, but you have to create opportunities and ask.

I agree with Thomas S. Monson when he said "In reality, we are all travelers - even explorers of mortality", and taking risks where rejection is a real possibility is necessary to personal exploration and conquest.

<comment><USER5> Without asking, you are already living the path of the "no". If you realize that, it becomes a lot easier to change your mindset.

I'd even say it's not about being hardened by rejection, but rather standing up for yourself. Speak up. Dare to ask something. Dare to want something. Then learning that taking risks increase your chances of getting a "yes". The famous "you lose 100% of the shots you don't make" :)

<comment><USER6> Very insightful! I have long been trying to integrate the "You miss 100% of the shots you don't take." way of thinking (this is the Gretzky quote, btw :)), but I never fully reasoned it out all the way: Not only am I passing up the chance, but I am receiving the same result as if the chance resulted in failure. In this way I now realize avoiding rejection is the same as being rejected. Powerful stuff.

Edit: Another anecdote that pops into my mind is Edison needing 10,000 tries to make the light-bulb. This means he failed 99,999 times. But we remember him for the success he had. (Queue pedants correcting the details of this anecdote ;))

</comment></comment></comment></comment><comment><USER7> The ideas brought up on that site are frequently gamed in the pickup scene.

 _Crash 'n' Burn_ is the practice of trying impossible sets (ie. girls) with the assumption of being rejected. If you are sure you will blow it big time, you can as well _do_ and learn something from it.

Another tangentially related technique against rejection angst is the _Three Seconds Rule_ which mandates you talk to some stranger before thinking for too long (which is really a different, short-term approach to the Rejection Therapy game).

</comment><comment><USER8> Did anyone else glance at this, think "rationality level too low, I cannot trust anything these people say" and give up?

<comment><USER9> I don't understand but it sounds interesting. Could you explain?

</comment></comment><comment><USER10> "Anybody who ever built an empire, or changed the world, sat where you are now. And it's _because_ they sat there that they were able to do it."

<comment><USER11> There might be some truth to that statement in general, but in _Up In The Air_ , usually it was obviously hollow, pacifying bullshit. (The guy with the cooking aspiration was an exception.)

</comment></comment><comment><USER3> Great idea. Hopefully the site will flourish and avoid turning into a spammy front for self-help products. It's a shame the content of the suggestion cards is only available by purchasing the physical cards.

<comment><AUTHOR> I'm working on a podcast where I personally execute all the suggestions on the cards, then discuss the outcomes.

So yes, the suggestions on the cards will be made available without purchase, via the podcast.

Thanks for the valuable input Zach!

<comment><USER2> Could you email me your list of rejection ideas? [my username] at gmail dot com

I promise not to post them in this thread, and if you do I will reply to your comment saying thank you. You could think of it as a publicity stunt.

<comment><AUTHOR> No.

Rejection isn't so bad, is it?

Seriously though, I'll do you one better. Email me a mailing address and I'll send you a physical copy of the suggestion cards absolutely free.

I can only do this for you though (I'm broke).

<comment><USER2> Mailing address emailed! Thanks in advance!</comment></comment></comment></comment></comment>


Has anyone had experience using Founder2be? - <AUTHOR> http://www.reddit.com/r/startups/comments/rjnx8/has_anyone_had_experience_using_founder2be_are

<comment><USER2> I did check the reply on reddit and on my blog[http://miemagazine.com/2011/09/06/how-i-got-some- cofoundersf...](http://miemagazine.com/2011/09/06/how-i-got-some- cofoundersfounder2be-com/) . I got some guys for my project www.ziliot.com</comment>


Announcing the Twitter Trust and Safety Council - <AUTHOR> https://blog.twitter.com/2016/announcing-the-twitter-trust-safety-council <comment><USER2> I'm sorry, but a council with more than 40 member organisations sounds a lot like lip service rather than anything substantive. How will this council do anything? The post explains why they need one, and says who they are, but doesn't say what they'll do apart from "tapping into their expertise".

</comment><comment><USER3> I don't see the ACLU or any other organization dedicated to freedom of expression on that council. This doesn't bode well for Twitter's support of free speech.

<comment><USER4> Free speech isn't aligned with their business model and their biggest share holders are not so keen on it either. Not saying I agree with it, just stating that it's on par.

<comment><USER5> For years, Twitter promoted itself as paragons of free speech even calling itself the "free speech wing of the free speech party." [http://www.theguardian.com/media/2012/mar/22/twitter-tony- wa...](http://www.theguardian.com/media/2012/mar/22/twitter-tony-wang-free- speech)

It looks like they have turned their backs on that. I don't know how to see it otherwise.

</comment></comment></comment><comment><USER6> And what happens when "more than 40 organizations and experts from 13 regions" aren't in agreement with each other?

This seems like hand-waving; and if it isn't, look what happened when reddit tried to move away from being an anything goes environment

<comment><USER7> I assume some of those organizations and experts are more equal than others.

</comment></comment><comment><USER5> For some perspective of how Twitter has changed over the years, particularly their relationship to free speech, here is Vice's History of Twitter's rules: [http://motherboard.vice.com/read/the-history-of-twitters- rul...](http://motherboard.vice.com/read/the-history-of-twitters-rules)

</comment><comment><USER8> It's a worthy goal, I wish the name didn't sound so... I dunno, Orwellian?

</comment><comment><USER9> > On Twitter, every voice has the power to shape the world.

> Grassroots advocacy organizations that rely on Twitter to build movements > and momentum

> ensure that we are enabling everyone, everywhere to express themselves with > confidence

Then why are they so eager to remove accounts of people they don't like or who speech they don't like. Weren't they bragging just last week about removing 125000 "terrorist" accounts.

</comment><comment><USER10> Detecting and hiding abusive tweets algorithmically is NOT THAT DIFFICULT. These methods are employed widely by Facebook and Google. It's baffling to me that Twitter doesn't do the same thing, and is willing to burn money on pointless bureaucracy like this.

Remember the "Other" inbox in FB? Twitter abuse could have been a footnote on par with that. Instead it's the only thing anyone talks about (deservedly so).

<comment><USER11> When you say something "algorithmic" is "NOT THAT DIFFICULT" on HN, I feel like you shoulder some obligation to explain some of the algorithmic approaches. You know, if it really is "not that difficult".

That sure as shit sounds like a more interesting discussion to have than a semantic debate about free speech.

<comment><USER10> A regression model trained from existing spam and abuse report data to twitter would cover ~99% of terrible at-replies on twitter.

Look how effectively a hacked-together script that just uses a couple of heuristics like GGAutoBlocker works. [1]

Something a little more principled, trained with unfettered access to user data, would effectively kill the problem. IMO.

[1] [https://github.com/freebsdgirl/ggautoblocker](https://github.com/freebsdgirl/ggautoblocker)

<comment><USER12> Your understanding of what is difficult is wrong because you're looking at it as a detection problem when it's actually a false positive problem.

See: [http://laughingmeme.org/2011/07/23/cost-of-false- positives/](http://laughingmeme.org/2011/07/23/cost-of-false-positives/)

<comment><USER10> That is an overly simplistic problem description.

It's trying to identify spam/abuse in a universal way, which is a much harder (and barely coherent) problem, and then delete those tweets which makes the cost of a false positive much higher.

What's far more feasible is to compute the probability a given user would like to see a specific tweet, and then have those below a certain threshold hidden (IE, don't ping the notification tab for it). This is actually a very well- studied problem, and the massively parallel ML infrastructure it would take exists. Eg, [http://www.datanami.com/2014/07/17/inside-sibyl-googles- mass...](http://www.datanami.com/2014/07/17/inside-sibyl-googles-massively- parallel-machine-learning-platform/)

</comment></comment></comment></comment></comment><comment><USER13> It's amazing that Twitter seems to think pouring effort and bureaucracy into "gentrifying" its userbase is somehow going to turn around the service. Twitter has a _major_ problem, but it's not cyberbullying or radical activists; it's that the product is super stale and they haven't innovated in years (ever?).

Not killing their ecosystem would have returned far more value to Twitter than becoming a platform with a hardcore political bias. Their inaugural members go so far as to include Feminist Frequency.

<comment><USER14> > Their inaugural members go so far as to include Feminist Frequency.

Why is this going "so far"? Anita Sarkeesian has been the source of a bunch of online (and offline) harassment, and has been outspoken against it, especially on Twitter.

<comment><USER15> > Why is this going "so far"? Anita Sarkeesian has been the source of a bunch > of online (and offline) harassment, and has been outspoken against it, > especially on Twitter.

Except of course that she's cried wolf on many occasions (going so far as to /invent/ threats and then pretend that she called the police about it and they advised her to tweet about it) and in general has _incredibly_ extreme views (which IMO don't hold up to even cursory scrutiny, but that could be my bias showing). The point is that a "trust and safety" council should also protect people from lynch mobs created by people like Feminist Frequency, right? So maybe we should get thunderf00t (not going to happen) to be part of this council?</comment></comment></comment>


New Heroku Dashboard and Metrics now in Beta - <AUTHOR> https://blog.heroku.com/archives/2014/8/5/new-dashboard-and-metrics-beta

<comment><USER2> Hey all. I'm one of the designer/engineers responsible for the new metrics tab.

I wasn't part of the selection process regarding Ember, which is the framework powering the entirety of dashboard-next. My personal feelings towards it are pretty ambivalent; as a python guy, I find it tastes a lot like ruby/rails (lots of magic). That being said, Heroku is a growing org, and a lot of people were involved in building out the new Dashboard — the strong conventions of Ember do make it easier for large teams to work together without multiple styles creeping into your codebase.

There's still a lot we'd like to do with Heroku Metrics, but we had to draw the line somewhere and ship a v1 that fulfilled a value proposition. Enjoy it, do (please!) give us feedback, and we'll iterate our way to yet more visibility into your app's behavior.

<comment><USER3> Nice work. Are the metrics collected using lumbermill?

[https://github.com/heroku/lumbermill](https://github.com/heroku/lumbermill)

<comment><USER2> Yes! I hope we'll have a engineering blog post covering our metrics pipeline soon.

</comment></comment></comment><comment><USER4> It's kind of strange that Angular seems to have more developer mind share but then when startups are choosing something beyond Backbone, they seem to go with Ember (e.g. this dashboard, Twitch, Vine, Square). I myself prefer Ember but it's still strange.

<comment><USER5> Why do you prefer Ember? For me the killer feature of angular is ability to build custom element (directives). Backbone + React is a good competition, but what about Ember?

<comment><USER4> Ember has components which are very similar.

<comment><USER6> I use ember full time now but I have done projects in angular. IMHO the directives were by far the cruftiest part of the experience, they had their own structure which didn't really fit with the rest of the framework. This in part because of the lack of a strong object model to leverage. I saw the same problem with knockout.js (although the bindings help matters a little). To date ember is the only framework I've used where it didn't feel like there was something fundamental missing.

</comment></comment></comment></comment><comment><USER7> Pretty cool! How do I add a favorite app so that it shows up in favorites? Or is that not functional yet?

<comment><AUTHOR> Hover over your app, on the right hand side you should see the "Add to favorites" star.

</comment></comment><comment><USER8> Any way I can see a preview without getting a heroku account?

<comment><AUTHOR> Yes, it's in the blog post here:

[https://blog.heroku.com/archives/2014/8/5/new-dashboard- and-...](https://blog.heroku.com/archives/2014/8/5/new-dashboard-and-metrics- beta)

<comment><USER9> Thanks. We changed the url to that (from [https://dashboard- next.heroku.com](https://dashboard-next.heroku.com)) because it's more explanatory.

</comment></comment></comment><comment><USER10> Much, much better than the previous version!

</comment><comment><USER11> Interesting that they have their own metrics now. Before they encouraged new relic.

<comment><USER12> this seems to be a visualization of data already present in the logs (behind some labs addon IIRC), but while useful it's a far cry from what new relic does (i.e. insights into _what_ is taking time and why, not just how much/when).

<comment><USER13> The Metrics displayed in the new Dashboard focus on the characteristics of app execution on Heroku, rather than internal instrumentation of the application. Hopefully that's useful in providing more visibility into tuning applications on the platform.

</comment></comment></comment><comment><USER14> Awesome! Congrats Heroku team!

</comment><comment><USER15> It's like, slightly better than it used to be. But let me take this opportunity to evangelize for Ninefold. [http://ninefold.com](http://ninefold.com)

Ninefold is, from my experience, everything you'd ever want from Heroku and much more (which in turn means much less). Far more simple UI and implementation, and extremely passionate and responsive dev team.

Please try out Ninefold if you've ever said "fuck you Heroku" like I have so many times.</comment>


Google buying Twitter predicted to follow Microsoft's move for LinkedIn - <AUTHOR> http://www.marketwatch.com/story/google-buying-twitter-predicted-to-follow-microsofts-move-for-linkedin-2016-06-14 <comment><USER2> If this were to actually happen I would consider it bad news for independent journalism and dangerous for some political viewpoints, even in the United States. Google already owns one of the largest social media platforms: YouTube. Both Twitter and YouTube have pretty large political significance and being owned by the same company could be an issue. Especially as we have seen some persons lose their verification badges (on both YouTube and Twitter) for allegedly being on the wrong side of the political spectrum. The idea of this makes me uneasy.

<comment><USER3> I agree with you in principle however as far as I can tell people are being banned for what is essentially hate speech, i.e. speech that incites to violence. That is not the same as banning for political views - Google and Twitter are under no obligation, not even an ethical obligation, to allow that stuff on their sites.

<comment><USER4> Hate speech can be very subjective. Like disagreeing with a common view point. If one company can control what is politically correct. There will be less freedom

<comment><USER3> Mere "political incorrectness" does not rise to anywhere near the level of actual hate speech. It's the difference between advocating against abortion or even protesting abortion clinics, and actually calling for the assassination of abortion doctors.

</comment></comment></comment></comment><comment><USER5> If Google were to buy Twitter, they wouldn't do it because they want Twitter's content or community, they'd do it to merge Twitter's single-digit display ad share with theirs and defensively eliminate a future competitor. I feel they have no more interest in Twitter than does Facebook.

Twitter acquired Crashlytics, and they released Fabric.io as a "Twitter Inside" SDK to entice app developers to Twitter integration and its MoPub ad network. This is their big push to beat out Facebook and live inside every app, but Google already plays this game, and Twitter isn't yet a huge threat to them. Either Google or Facebook could buy them defensively, but they need not at this time. (I feel that Microsoft's purchase of LinkedIn was also defensive, but that's a different story.)

Verizon, on the other hand, could buy Twitter. They bought AOL to enter the content creation market, and they could cement their share. Or more likely, they could instead buy Yahoo, and with it Tumblr, on the cheap.

Amazon could buy Twitter. They bought Twitch, and they could use Twitter to expand their social presence, logged-in time, and display ad share.

Yandex could buy Twitter. They would break into the US and Japanese market in a huge way. The Fabric.io SDK would be a perfect fit to drive more engagement from apps to their properties.

</comment><comment><USER6> This is something I believe Google should have done way before. They need an external hand to make a footprint in social media and Twitter might be the perfect bet.

<comment><USER7> I don't think it would be possible before since Twitter thought that it will be super-profitable.

Only now they realised that they have no real model for making actual money. It's the perfect time to buy.</comment></comment>


Ask HN: Parenting App Idea - <AUTHOR> for those of you with kids, would use like an App your kids can use to interrupt you while you're working. i.e. they want your attention and you want to keep working. The way this app would work is like Uber with surge pricing. You set your price per interruption and the kids learn when it's $5.00 per interruption I'll just wait for that surge price to go down to $1.00 per interruption unless it's really really important. <comment><USER2> Would that be deducted from their allowance? I guess this would rather alienate my kids from me, as they'd try to save money.

</comment><comment><USER3> Clever idea, but kids don't care and as a parent your always on call.

<comment><AUTHOR> not always the same level of on call. Sometimes I'm in my office and if they have a problem their mom is there. But they keep coming into my office and bugging me. And yes, I like the interruptions sometimes, they are great kids, but sometimes I wish I could say, for the next 2 hours, surge pricing!

</comment></comment><comment><USER4> I'd much rather an app that had age appropriate videos for kids. I have an 8 year old that's too old for youtube kids and to young for youtube.

<comment><AUTHOR> should the on-ramp to adulthood be a slow merge lane, or a drop you in the middle of 65 MPH traffic the day you turn 18?

<comment><USER4> I guessing my kids won't even drive. autonomous vehicles and all.</comment></comment></comment>


Re-checking the ReactOS project - a large report - <AUTHOR> http://www.viva64.com/en/b/0192/

<comment><USER2> See previously John Carmack on the importance of static analysis [http://www.altdevblogaday.com/2011/12/24/static-code- analysi...](http://www.altdevblogaday.com/2011/12/24/static-code-analysis/) <https://news.ycombinator.com/item?id=4543553>

</comment><comment><USER3> From the title, I was expecting to read a review of ReactOS looking at whether it would be viable as a day-to-day OS. This leaves me a little bit disappointed.

<comment><USER4> Okay, I'll write that review:

No.

It's unstable, it lacks support for any filesystem but FAT32, it lacks meaningful driver compatibility with Windows, it doesn't support multiple users, it doesn't support major APIs like thread pools/WMI/WDDM, its TCP stack is nascent, there is no SMB support...

I want this project to succeed, but it needs a _lot_ of love to get there.

</comment></comment><comment><USER5> One of the NULL dereference diagnostics looks spurious:

      DomainName = &PolicyAccountDomainInfo->DomainName;
      SidNameUse = (PolicyAccountDomainInfo != NULL ?
                    SidTypeGroup : SidTypeUser);

This does not, in fact, dereference PolicyAccountDomainInfo, but simply performs some pointer arithmetic. It's bad style, agreed, but should be entirely harmless to execute.

<comment><USER6> Doing _anything_ with invalid pointers is UB in C++. It might be fine now, but it might not work in 5 years, when compiled with a newer compiler.

E.g. a sufficiently clever, malicious compiler could reason that because you are doing pointer arithmetic involving PolicyAccountDomainInfo->, the pointer must be valid and then it might get rid of the null check and branch in the second line. I remember not so long ago there was a vulnerability in Linux kernel because of exactly this kind of "malicious" optimisation performed by gcc.

<comment><USER5> As others have commented, the Linux incident involved an actual dereferencing of the pointer.

The way I read <http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1570.pdf> top of page 89 and footnote 102 on that page is that this is NOT undefined behavior in C, and though I haven't checked the corresponding language in the C++ standard, I would think it's not undefined in C++ either.

</comment></comment></comment><comment><USER7> Are there good low-end tools for C/C++? I've been doing iOS development and I recently decided to write some of my code in C++ so it would be portable to Android and Blackberry.

<comment><USER8> If you've been working on iOS you already have it installed <http://clang- analyzer.llvm.org>

<comment><USER7> Yes, Xcode with clang is really nice. After reading Carmack's blog, however, I'm sure more can be done. He actually like's Microsoft's compiler. Wonder if it's worth running code through it?

</comment></comment></comment><comment><USER6> Hmm, don't ReactOS developers use IDEs? Or do C++ IDEs lack advanced static analysis features?

<comment><USER9> It depends. Some of us use Visual Studio. Some of us use gedit.

<comment><USER10> If you do use Visual Studio, it would be interesting to know what do you think about its static analysis (or absence of)

</comment></comment></comment><comment><USER11> Interesting project. Is it fairly easy to get up and running as a contributor?

<comment><USER9> Yes! There's a single build environment (RosBe) to download (for both Windows and *NIX), then you just need to do an svn checkout (svn is included in the environment), and then make a patch and submit it for some bug.</comment></comment>


Ask HN: Examples of well-written/accurate front-end dev. job descriptions? - <AUTHOR>

<comment><AUTHOR> I think this UX Engineer role from Google is actually pretty solid.

[https://www.google.com/about/careers/search#!t=jo&jid=45100&](https://www.google.com/about/careers/search#!t=jo&jid=45100&)</comment>


Outsourcing Core Business - tomh- http://swombat.com/2011/4/5/outsourcing-core-business <comment><AUTHOR> But what is your core business?

Most startups claim to be technology companies. I think 90% of these are tech- based companies as opposed to technology businesses. The difference being the latter kind is more like Intel or VMware. If you are tech-enabled, then what is your core business? Is it the software you write?

<comment><USER2> For most tech-enabled subscription businesses, the core business is support. Your technology is easily duplicable, so your key differentiator has to be how you deal with customers, and how quickly.

Sales is core too, but sales is also a support task - you talk the customer through how you can help them and what they'll need to do in order to use your product. Educating someone into a sale is pleasant for both sides.

Doing support is time consuming, so make sure you factor that into your prices. Can you really afford to support your product for $9/month? Probably not, your time is worth more. Charge more than $9!

<comment><AUTHOR> Is it then safe to say you can outsource software development and focus on customer acquisition and support?

<comment><USER2> No, only that once you have your core features in place, building more won't improve your product and won't win or keep customers. Meanwhile, as a non-tech business, your competitors can easily copy the features you built, so how do you win? In the long-term you have to beat them on sales and support, and outsourcing that to someone without a deep knowledge of the product (as in the article) makes that hard.

</comment></comment></comment></comment><comment><USER3> The best entrepreneurs, from what I have seen, are the people that want to and must be involved in all core business responsibilities. The passion and persistence required to create something demands this.

However, I believe there are exceptions to this, such as design. If you have great product vision, technical ability, understand your users and you are infinitely passionate about the product you are building, I see no problem with outsourcing (at least initially) to 99designs or similar sites.</comment>


Show HN: Topic-based 1-on-1 chat - <AUTHOR> https://convabout.com/ <comment><USER2> Looks interesting, I checked out a couple but didn't see much in the way of response. Probably a chicken/egg issue. Nobody to chat with yet, so no reason to stay and chat, etc.

<comment><AUTHOR> Did you click "start"? In that case _you_ are meant to respond. Again: 1on1 chat.

<comment><USER2> yeah, I found an existing chat and responded. I waited a minute and didn't see a reply so I moved on. Maybe I'm just impatient. If the intent is for not all chats to be realtime, maybe something to remind folks to check back may help until you hit critical mass. If not, maybe some kind of indication that the other party is present and waiting for a response.

<comment><USER2> It would also be nice to be able to easily leave a conversation so it doesn't fill up the 'my conversations' window. The conversation could then go back to the main list to be fulfilled by someone else maybe.

<comment><AUTHOR> Aha. Maybe a flag both on the index page and in the chat window that a user is offline? And an option to leave a conversation even if a responder wrote messages but didn't get any from an author yet? Then this conversation goes back to the index page.

<comment><USER2> Yeah that'd be cool. There could be other circumstances where you'd want to be able to leave a conversation as well-- abusive, bad partner, etc. Anyway again- nice work.

<comment><AUTHOR> Good point. Thank you!

</comment></comment></comment></comment></comment></comment></comment><comment><USER3> No idea how it works. Click on "start" and nothing happens.

<comment><AUTHOR> Start=start conversation. It redirects you to the chat window. Should I name the button "start chatting"?

<comment><USER3> It doesn't seem to work in my browser:

>(index):1 Uncaught (in promise) null

<comment><AUTHOR> You're right. Fixed now. Thank you!</comment></comment></comment></comment>


JSON Lines - <AUTHOR> http://jsonlines.org/examples/ <comment><USER2> This is not a good replacement for CSV. It is better to just use regular JSON for that.

If you have a file containing this:

      ["Name", "Session", "Score", "Completed"]
      ["Gilbert", "2013", 24, true]
      ["Alexa", "2013", 29, true]
      ["May", "2012B", 14, false]
      ["Deloise", "2012A", 19, true]

And it gets chopped in half (a problem that is not that uncommon), you will get this:

      ["Name", "Session", "Score", "Completed"]
      ["Gilbert", "2013", 24, true]
      ["Alexa", "2013", 29, true]

Which is still valid. This will cause missing data rather than a clear error message. May and Deloise may end up not getting their scores and there's a good chance nobody will notice.

By contrast when the file is represented as regular JSON:

      [
        ["Name", "Session", "Score", "Completed"],
        ["Gilbert", "2013", 24, true],
        ["Alexa", "2013", 29, true],
        ["May", "2012B", 14, false],
        ["Deloise", "2012A", 19, true]
      ]

(not a completely ideal representation, but you get the picture)

Chopping _this_ in half will give you a clear error, forcing the developer to recover from it before continuing.

Same principle as strong vs. weak typing.

Individual JSON snippets in lines are a good replacement for plain text-only logs of indeterminate length, however.

<comment><USER3> Regular CSV also has that "problem". Being able to cut the file into pieces is actually really useful when you want to, for instance, process each piece on a different computer. Line-delimited JSON preserves that advantage.

<comment><USER4> Exactly. If JSONL takes off, I'd be glad to use it instead of YAML for streaming serialization/deserialization purposes, if only because JSONL is more space efficient.

</comment></comment></comment><comment><USER5> There is also NDJSON (aka Newline-delimited JSON): [http://ndjson.org/](http://ndjson.org/)

(disclosure: I'm the author of the Haskell ndjson-conduit library)

<comment><USER6> Any thoughts on merging the two efforts. They seem similar enough that you could focus effort on one of them.

<comment><USER7> Interesting idea. If I'm not mistaken, JSON Lines looks like a subset of NDJSON?

</comment></comment></comment><comment><USER8> I wonder, why even bother with wrapping rows in array brackets and not do it implicitly.

        ["Name", "Session", "Score", "Completed"]
        ["Gilbert", "2013", 24, true]

would be

        "Name", "Session", "Score", "Completed"
        "Gilbert", "2013", 24, true

Which is, well, more or less just CSV. This should work with objects too:

        "name": "Jane", "key": { "nested": "object" }, "foo": ["bar"]

Or mixed:

        "Foo", { "fnord": 23 }, true

<comment><USER6> One benefit with each line being valid JSON is that writing a reader/generator for this format is fairly simple. One can use existing JSON libraries as-is, with some extra consideration to the line-like nature of this format.

For the above to work you'd have to use a custom JSON-parser when reading the lines.

\</comment></comment><comment></comment><comment>

Could also be mentioned that JSONLines-like formats are already pretty common in log-files, database exports etc. So this is more about giving it a name and standardizing it. Which I think is great!

<comment><USER9> >> For the above to work you'd have to use a custom JSON-parser when reading the lines.

Not necessarily. You could just re-add the square brackets before passing the lines to the JSON parser.

That would make more work for the machine, because you'd probably have to copy the whole string. But any time you can make less work for the human by making the machine work a little harder, it's usually a win.

</comment></comment><comment><USER10> Heh. I wrote a post about this last year (and I'm not the first) [http://willi.am/blog/2014/07/16/storing-data-as-newline- deli...](http://willi.am/blog/2014/07/16/storing-data-as-newline-delimited- json-entries/)

The main unique thing here is the proposal that tools like Excel should support this format. That's definitely an interesting idea. Biggest challenge is that the JSON objects can be nested, so table tools would need a way to handle that. Still, interesting

<comment><USER11> Excel already supports nested data in plain XML (I say "plain", because Excel also supports Microsoft-specific XML schemas), so it might not be that hard to apply the same parser / deconstructor concepts to nested JSON structures as well

</comment></comment><comment><USER12> I don't see what this improves over standard JSON. Just add a comma at the end of each line and wrap the whole Lines file in brackets, and you have a valid JSON file. You can petition spreadsheet makers to support this subset of JSON files, while still letting any general JSON program read and write them.

As for CSV and TSV, they're a dead-simple format to read and write, and you can enforce a standard way to escape special characters if you aren't dealing with arbitrary user-submitted files. With numeric data escaping isn't even necessary, and if you have to deal with strings that might themselves contain special characters, the ASCII control characters 0x1E and 0x1F work well as alternatives to comma/tab and newline.

I know Python has a csv module, but I've just gotten used to writing these snippets:

        def tsv_write(path, headers, data, sep='\t', end='\n'):
            with open(path, 'w') as file:
                file.write(sep.join(map(str, headers)) + end)
                for datum in data:
                    file.write(sep.join(map(str, datum)) + end)

        def tsv_read(path, headers, sep='\t', end='\n'):
            with open(path, 'r') as file:
                for line in file:
                    yield line.rstrip(end).split(sep)

(That tsv_read function only works if end == '\n'. Here's a somewhat- inefficient general-purpose alternative, although I've never needed it.)

        def read_records(file, end='\n'):
            if end == '\n':
                for line in file:
                    yield line
            else:
                record = []
                while True:
                    c = file.read(1)
                    if c:
                        record.append(c)
                    if c == end or not c:
                        yield ''.join(record)
                        record = []
                    if not c:
                        break

        def tsv_read(path, headers, sep='\t', end='\n'):
            with open(path, 'r') as file:
                for record in read_records(file, end):
                    yield record.rstrip(end).split(sep)

<comment><USER13> The advantage over standard json is that this is trivial to parse incrementally, i.e. for a dataset that is too big to fit in memory.

I also wouldn't recommend you to reimplement a functionality that is already part of the standard library, especially because the standard library function is going to be much faster because it's implemented in C.

<comment><USER14> Actually, the approach with commas and a wrapped brackets are equally trivial to parse incrementally.

Just start by erasing the open "[", and then read one row at a time, just like you would do without the "[" and comma.

<comment><USER15> I would say any tool that outputs lists of things in JSON and there is reasonable suspicion that the output may be large, should do this. I.e. do one-thing-per-line with opening and closing brackets on separate lines. It is quite annoying to find oneself with huge dataset in JSON which was produced by a naive generator that doesn't do that - when you need to extract something like "first N elements that satisfy condition X" it is annoyingly more work that it should be. By doing this people that don't care still get valid JSON, people that do care save some time.

</comment></comment></comment></comment><comment><USER16> It's good that someone gave this format a name. Although it's not called anything special, but Go can read this format easily. See the Decoder example [1].

If you don't actually need a stream, I think it's a bit nicer to write a special header at the beginning and make the whole file a single JSON object:

      {"header": {"title": "An example", ...},
       "rows": [
      ["row1"],
      ...
      ["rowN"]
      ]}

[1] [https://golang.org/pkg/encoding/json/#example_Decoder](https://golang.org/pkg/encoding/json/#example_Decoder)

</comment><comment><USER17> Been doing this for log files for years now. It's pretty convenient and simple, but I'm not sure that any end user would want to see this -- they're just going to want CSV.

</comment><comment><USER9> I came up with an idea for a CSV replacement that's half-joke, half-serious. It's called XML Separated Values (XSV). It's meant to be a 1:1 drop-in replacement for CSV. It solves the ambiguity and encoding issues of CSV by recycling XML's solutions. It has basically two tags: <c/> and <n/>, for comma and newline, respectively.

Unfortunately, it needs a root element to be valid XML, so wrap the document in <xsv></xsv>. Also, add an XML declaration.

Example:

      <?xml version="1.0" encoding="UTF-8"?>
      <xsv>one<c/>two<c/>three<n/>1<c/>2<c/>3<n/></xsv>

(Whitespace is passed-through, therefore it's significant.)

This solves the ambiguity problems of CSV without complicating CSV by introducing things like data types (I think the virtue of CSV is its simplicity), and it should be extremely simple to write parsers based on either SAX or DOM (, etc).

Also, the use of XML provides unobtrusive places to insert application-defined metadata, for example:

      <xsv columns="3">

This remains backwards-compatible with parsers that ignore attributes.

<comment><USER18> CSV is human readable, how readable is left to the data.

XML tries to be human readable, but you will need knowledge of elements or "tags", meta data.

JSONLines also tries to be human readable, imo it does a better job, compared to XML.

But not until Excel of LibraOffice implement a JSONL, or XSV reader/writer.

Will CSV be the de-facto multi standard.

<comment><USER19> In my experience XML is easier to read than JSON. Guess it depends on the complxity of the structure.

<comment><USER9> I think one predictor of readability is the ratio of markup to data. XML gives you some flexibility there, but XML-based formats tend to have very verbose markup, which overwhelms the data.

To cut back on markup in XSV, I used self-closing tags for commas and newlines. It would have been more conventional to use something like <row><column>...</column></row>. That has some advantages, maybe, but it doubles the number of tags (and it introduces things that are not strictly 1:1 with CSV, which could lead to ambiguity).

</comment></comment></comment></comment><comment><USER20> > no standard column separator

Commas are the standard column separator in CSV files, its in the name.

<comment><USER21> I'll give the author the benefit of the doubt; Excel appears to export by default using tabs as delimiters, not commas: [https://support.office.com/en- za/article/Import-or-export-te...](https://support.office.com/en- za/article/Import-or-export-text-txt-or-csv- files-5250ac4c-663c-47ce-937b-339e391393ba#bmchange_the_delimiter_that_is_used_in_)

<comment><USER22> It actually depends of the locale of the system.

If you are on a French windows, it will use semi-colons as separators, encode in window cp1252 (close to latin1 but not exactly) and use CRLF (\r\n) as line returns.

But then if you use Excel for Mac 2008, Excel will generate CSVs in MacRoman (old OS9 encoding) and use CR (\r) as line returns.

Long story short Excel is not even interoperable with itself when you use CSVs. (Granted there is an advanced import tool, but it's a nightmare for the average user).

[https://fr.wikipedia.org/wiki/MacRoman](https://fr.wikipedia.org/wiki/MacRoman)

<comment><USER20> Wow, never realised it was that complicated. Although the locale dependant separators make a lot of sense.

</comment></comment></comment></comment><comment><USER23> This is great. Good to see it named and an extension proposed. Many tools assume .json means a single JSON expression. It would be nice to have GitHub allow one JSON expression per line.

I see someone already created a GitHub linguist issue for it: [https://github.com/github/linguist/issues/2217](https://github.com/github/linguist/issues/2217)

</comment><comment><USER24> I prefer JSON text sequence ([https://tools.ietf.org/html/rfc7464](https://tools.ietf.org/html/rfc7464)). Self healing in the event of any corruption/parser bugs is nice, JSON lines seems like it will have issues finding a valid object start if your data ever contains new lines and you have some issue parsing one of the objects.

</comment><comment><USER25> Speaking as someone who just (today) was migrating game entity data from JSON to CSV, the main reason I can see to use CSV so you can edit in Excel/OpenOffice/Libre/etc.

If there was a decent visual tool (or chrome extension etc) for editing data in a format like this new one, I'd probably jump on it, but otherwise it seems like anyone currently using CSV might as well stick with it.

</comment><comment><USER26> Aka how pretty much everybody stores logs already.

</comment><comment><USER27> Mongodb mongoimport/mongoexport already uses this format. It would be nice to get it formalized with a recognised file extension and mimetype.

</comment><comment><USER28> it's also worth mentioning JSONH for compact represenation of CSV data in json:

[https://github.com/WebReflection/JSONH](https://github.com/WebReflection/JSONH)

</comment><comment><USER29> I'll raise this one, I use JSON lines for version management.

Each line is a version, with only the changes.

This means you can't delete though, but for blockchain-like distributed systems that's almost a feature.

</comment><comment><USER30> I agree that CSV doesn't have a standard, but for all practical purposes - wouldn't most people agree that the Excel dialect is the standard CSV dialect?

<comment> X-Istence Not when you see some of the CSV data you have to parse from different vendor's implementations. Especially when writing reporting tools.

I've seen so many different dialects, even from the same vendor... that's the worst part.

</comment></comment><comment><USER31> Problems with CSV parsing? Just use TSV. I've never had a parsing problem with TSV.

</comment><comment><USER32> This is how bulk operations in elasticsearch work. I'm a fan.

</comment><comment><USER33> Nop. Why? You are breaking JSON. Why not a coll of objects? To save bytes? Seems like one of your main motivations is 'csv lacks standards' and then you go break a standard. In any case call it something else but no JSON.

<comment><USER34> It's not breaking JSON...it's explicitly stating a new format, free-text delimited by \n and \r\n, wherein each line must be valid JSON.

This is fine.</comment></comment>