Evasi0n iOS 6.x jailbreak - <AUTHOR> http://evasi0n.com/?o=1 <comment><USER3> So you guys use a closed source program from 'hackers' to be able to fully use your phone? So you have all the disavantages of paying for something and all the disadvantages of using some keygen binary to use as it was pirated.

and i'm worried about what lies in the binary blob of my phone's boot1 and radio driver....

<comment><USER4> I'm sure you're not using a closed source binary anywhere in your day to day usage of computing devices.

<comment><USER5> Sure, but my laptop's graphics driver isn't preventing me from sharing a torrent while trolling^Wbrowsing HN.

</comment></comment></comment><comment><USER6> Some interesting strings from the binary. Hopefully there will be a write-up explaining the exploit in detail soon.

      /var/mobile/DemoApp.app
      Media/Recordings/.haxx/DemoApp.app/Info.plist
      Media/Recordings/.haxx/Library/Caches/com.apple.mobile.installation.plist
      Media/Recordings/.haxx/timezone
      Media/Recordings/.haxx/var/evasi0n/evasi0n

Edit: I also spotted a few references to Racoon, the VPN client which I think was abused in an earlier jailbreak as well?

<comment><USER7> Here is a more detailed analysis: [http://blog.accuvantlabs.com/blog/bthomas/evasi0n-jailbreaks...](http://blog.accuvantlabs.com/blog/bthomas/evasi0n-jailbreaks- userland-component)

The lazy binding is really interesting. There is lots of interesting stuff in the kernel, dyld, and libSystem. I would encourage you to have a look! You can do interesting things like run code before libSystem_init[1]

[0]:

<https://github.com/Apple-FOSS-Mirror/dyld/tree/master/src>

[https://github.com/Apple-FOSS- Mirror/Libsystem/blob/master/i...](https://github.com/Apple-FOSS- Mirror/Libsystem/blob/master/init.c)

[https://github.com/Apple-FOSS- Mirror/xnu/blob/master/bsd/ker...](https://github.com/Apple-FOSS- Mirror/xnu/blob/master/bsd/kern/kern_exec.c)

[1]: <https://gist.github.com/jevinskie/4615901>

<comment><USER6> Nice analysis. According to twitter, there's something more to it, though, there still seems to be some sort of kernel memory corruption exploit:

<https://twitter.com/kernelpool/status/298714209187921921>

</comment></comment></comment><comment><USER8> Shame it doesn't happen in the browser this time around. That PDF exploit last year was _slick._

<comment><USER9> If it was a browser exploit, wouldn't that put all users at a great risk, and also put more pressure on Apple to rapidly patch?

<comment><USER10> IIRC, the first jailbreakme.com used a PDF exploit that Apple patched REALLY fast, for the very reason you cited.

<comment><USER11> May have been TIFF?

<comment><USER12> Aye, I believe it was the TIFF exploit.

</comment></comment></comment></comment></comment><comment><USER13> Not worth it because the parallel jailbroken world instead of being something full of free and interesting things is poor and full of things that you need to pay for, but, 5 times the average app price on iTunes.

I'm not going to get instabilities and hard to upgrade devices just for SBSettings or alike.

Unfortunately too many people only jailbreak so that they can install software for free. That said it completely sucks that Apple does not allow me to be in control of what I want to install, including apps downloaded from a web site.

Also, a lot of people install SBSettings just because how freaking lame is the iOS algorithm to understand how bright the screen should be a given light level. Lame.

<comment><USER14> Please don't lump everyone wanting to JB into the same (laughable, to me) SBSettings or the even more presumptuous pirating use cases. For example, I'm in Canada, and my favourite music service is Grooveshark (Spotify is not available). The only way to get the Grooveshark iOS app is through jailbreaking.

<comment><USER13> Sure, too many people but definitely not everybody, and I just talk for what I see, that is, the population here in Italy that I can monitor directly (however in Italy there is a strong inclination for software piracy).

</comment></comment></comment><comment><USER15> Awesome! Does anyone know if SMS GV/Phone GV extension is working or not? I tried the tethered jailbreak on the 3GS and it wasn't working then. Only the Phone GV was working but it was slow.

I have an iPhone 5 now and hopefully the day ends early today so I can go home and jailbreak. Can't wait to get NCsettings and Google voice integration!

<comment><USER16> They aren't updated for iOS 6 yet, but I believe the developer is planning on updating them. You can email him for more information if you like - his email address is listed at the bottom of his website (<http://gvexts.appspot.com/>).

</comment></comment><comment><USER17> Here are my mirrors:

Windows: [http://asiekierka.pl/evasi0n-win-1.0-3c53ba10e2448d311b0f415...](http://asiekierka.pl/evasi0n-win-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.zip)

OSX: [http://asiekierka.pl/evasi0n-mac-1.0-3c53ba10e2448d311b0f415...](http://asiekierka.pl/evasi0n-mac-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.dmg)

Linux: [http://asiekierka.pl/evasi0n-linux-1.0-3c53ba10e2448d311b0f4...](http://asiekierka.pl/evasi0n-linux-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.tar.lzma)

<comment><USER18> Your sha for OSX does not match those listed on the site.

Also the size is different: 9.6M evasi0n-mac-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release (1).dmg 9.2M evasi0n-mac-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.dmg

I'm choosing not to open yours.

<comment><USER17> Odd.

<comment><USER19> I also checked the shasum by comparing 2 files.

The Mega link (Mac client) from the official web page: bd9fe1e58343a5c03295a975697de3e64e65b42c

asiekierka's Mac link: bd9fe1e58343a5c03295a975697de3e64e65b42c

Both file sizes are the same (9,690,941 bytes, 9.7 MB on disk). Maybe the discrepancy has to do with -a flag for 224, 256, 384, or 512 or something.

I think his mirror is safe. It's the one I used to successfully jailbreak my iPad 3. Now I can SSH into my iPad and change the hosts file for some ad- blocking and installed Flux for "easier on the eyes" nighttime reading.

I don't look forward to Apple's next update which will wipe out my jailbreak, so I don't want to customize things too much. Last time I did a jailbreak for my iPod Touch, it messed up an in-app purchasing mechanism for a game I was testing. After this experience, I dread that jailbreaks will cause unforeseen problems and prefer not having to customize every single little tech option. Still, I performed this jailbreak mostly to get ad-blocking in MobileSafari.

Actually, one snag I have: the iPad time was totally wrong, insisting that it was 8 hours ahead, and the General Settings wouldn't properly automatically update the time. Had to do a lot of fiddling with Location services and Date & Time to fix it.

Jailbreakin' ain't all that it's cracked up to be!

<comment><USER19> Another glitch: the volume controls don't work correctly. No matter what level your volume is at, if you press the down button just once, the volume will go completely to 0! You have to keep pressing up again to reach the proper level. Quite annoying when the volume is a function used so many times a day. Jailbreaking may allow you to get more features, but it usually brings a whole host of headaches, too. I'll probably just stick with the standard iOS system after the next update.

</comment></comment></comment></comment></comment><comment><USER20> The last time I jailbroke my devices they slowed to a crawl and crashed frequently. Felt like I had installed Windows on my iOS device. I'd much rather have my walled garden of reliable bliss.

I'll still donate to the cause however. Keep up the fine work gents! :)

<comment><USER21> It would have been the apps that you installed, not the jailbreak itself.

<comment><USER20> Oh it definitely was.

</comment></comment></comment><comment><USER22> "If the device is stuck displaying "patching kernel", you can press Power+Home for a long time to force a reboot." [1]

[1] <https://twitter.com/pod2g/status/298493685765648384>

</comment><comment><USER23> There's a Cydia app called BrowserChanger that lets you change the default browser (so clicking a link in Mail will open Chrome instead of Safari, for example). Is there a similar app that lets you change the default maps application?

<comment><USER16> MapsOpener makes Google Maps the default maps app - <USER2> <http://www.idownloadblog.com/2012/12/14/mapsopener/> for some information and a video.

<comment><USER23> Thanks!

</comment></comment></comment><comment><USER24> The last time I had a jailbroken phone was at the time of iOS 2, back then I didn't think it was worth it.

What has changed in the last four years?

<comment><USER25> I do like having OpenVPN and better wireless scanning/details.

<comment><USER26> What do you use for wireless scanning?

<comment><USER25> Sorry late response... One is WiFi Analyzer. Shows Channel, strength, encryption, ssid. Has a nice little great with signal strength along with min/max/avg. Not sure if you can get it from appstore now as someone else pointed out can get OpenVPN officially there now :)

Also... I'm curious as to why Apple hasn't added a SBSettings / NCSettings type feature as most other smartphones have toggle buttons in the "notification area" which make life a lot easier.

</comment></comment></comment></comment><comment><USER27> As a sidenote, I'm interested in hearing piracy numbers after this jailbreak. Is piracy still popular, or are people just jailbreaking solely for the mods?

<comment><USER28> didn't hackulous develop Appsync? That's the only simple method I knew of for transferring pirated apps from iTunes to the iDevice. Which is the main way the non technical users did it. Now that they've called it quits piracy will probably be drastically slowed down until there are new well known alternatives.

I could be totally wrong though, I don't own any iDevices, just basing my knowledge off helping people jailbreak their phones and pods over the past many years (except this last year of course..)

<comment><USER29> Try AppCake or vShare. Installous is dead.

<comment><USER27> I'm not interesting in pirating apps, I'm interested in either prevention and/or not selecting iOS as a games platform. Nothing has shaken me more than buying an iPad 3 off Craigslist because the dude couldn't jailbreak it to get free apps. (And he definitely did not come across as a technical person.)

<comment><USER29> It will always happen. The only known way to prevent this is to get your users to authenticate in your DRM service.

</comment></comment></comment></comment></comment><comment><USER30> The jailbreak community, who overall do very good things for users, are shooting themselves in the foot by not having the Cydia store work properly once you jailbreak --- the store is totally down, servers slammed. They had 6 months to prepare for this? why don't they correct it? people's first impressions (who have never jailbroken before) are that the process is terrible.

I am referring to the Cydia store by @saurik.

<comment><USER31> Look, I mostly do this because I find it meaningful. It pays miserably, and yet people always act like I'm making tons of money off of it, so a ton of people hate me and I don't even get to self-medicate by staring at a massive pile of cash. I've built a lot of really cool assets, but they (even the Cydia Store) are totally reliant on security flaws in something that is becoming more secure every day, so the work is even futile.

That doesn't mean, however, that I should be expected to perform miracles. I already work nearly every waking hour on things related to jailbreaking: "I'm giving er' all she's got, captain". For the record, by the way, here is what I'm up against today:

<http://cache.saurik.com/tinyimg/cydia6hits.png>

The game is also really difficult to predict. In this case, this is unlike any previous jailbreak, because they announced a specific time. I was not expecting them to announce a time: they have never announced a time before. I was not prepared for them to announce a specific time. What normally happens is there is a massive wave over the first few days while people find out about the jailbreak. Today, there were people watching a progress bar for hours until it hit 100%.

Meanwhile, you make it sound like it is really simple to take a payment transaction and licensing system, and just go "oh wow, we are doing over 10x the load? flip the switch boys!"... payment systems just don't work like that: this isn't some stupid web forum where you can play fast and loose with consistency (or even durability) to get more performance.

Also, six months to prepare? Seriously: you think I should spend six months while Cydia is losing money and there are no jailbreaks available--and there may never be a jailbreak available again--sitting around figuring out how to make a payment system scale infinitely so that during a small multi-hour long window it can shine?

Even in jailbreak-land, that is not the most important thing to be doing; one of my big time sinks this last half year was figuring out how to better deal with credit card fraud, for example. Leaving the world of payments, vendors are happier with more backend features, developers are happier with more Substrate improvements, and end users would prefer I make more tweaks or add things to WinterBoard.

<comment><USER32> This may be one of the most anticipated jailbreaks ever.. I'm not sure, even without announcing a specific time, you wouldn't get hammered. The moment the jailbreak went up, sites like reddit and hackernews would be on it like a flash.

You do great work, don't let the few smaller complainers get to you.

</comment></comment></comment><comment><USER33> Am I the only one getting a stream of NetDB errors (with the occasional HTTP 500 or 502)? I guess I'll just have to wait until... whatever server it's trying to connect to is up and running. I haven't had a chance to install a single package yet.

Edit: Popping over to r/jailbreak confirms it, I guess it'll be a while before things calm down. I just hope I can get f.lux before I have to go to work.

<comment><USER34> Confirming this as well... If you see this error, NetDB, Failed to fetch, etc., it is not any problem with your iDevice, Cydia, the jailbreak, or really anything at all. In fact, if you are seeing this right now on 2/4/2013, it most likely means everything is working properly!

The servers that provide the apps, the 'sources', are overloaded with people trying to download. Waiting until the servers are less loaded is the only solution, right now.

<comment><USER35> Me too.

</comment></comment></comment><comment><USER36> Is this a tether jailbreak or is it permanent?

<comment><USER16> It's an untethered jailbreak (allows the device to reboot on its own without using a desktop tool to help it boot), if that's what you mean by permanent. I probably wouldn't call it "permanent" since you can easily remove the jailbreak by restoring the device with iTunes.

</comment></comment><comment><USER37> Seems to be crashing on me instantly on 10.8 with my 3rd Gen iPad:

Application Specific Information: __* error for object 0x10fea00: pointer being freed was not allocated

</comment><comment><USER38> I haven't Jailbroke my iPhone for a few years. I did out of curiosity, but updating OS was just too much pain.

Been thinking what are the real benefits of doing that? Why? Is it customizing UI, adding more effects? Or is it just downloading apps outside app store? I think it brought value when we didn't have notification center etc, but still people doing it?

<comment><USER39> The big one for those of us with an AT&T unlimited data plan is tethering. They had to grandfather our plans in because they sold them to us as unlimited, but they will not offer tethering as they are trying to force all users off of those plans and on to the 5GB plan.

<comment><USER40> Do those apps actually still work on AT&T? I thought AT&T detected their operation and blocked them.

<comment><USER41> They still do work, PdaNet has an option to hide the tethering from AT&T. <http://junefabrics.com/iphone/index.php>

</comment></comment></comment></comment><comment><USER42> It worked for me, but now I am getting error "HTTP/1.1 500 Internal Server Error" in cydia

It looks like this also happened to people alot when the jailbreak for iOS 5 first appeared, so I am pretty sure cydia is just getting flooded with traffic from hoards of newly jailbroken users.

</comment><comment><USER43> Curious, does it work with the iPad Mini? [EDIT: Yes. <https://twitter.com/tomle12/status/298496235894095873>]

Also, how soon could Apple push out an update to kill this one?

</comment><comment><USER44> Definitely waiting until tonight to jailbreak. Last few times my phone got slightly screwed up when I jailbroke right after it came out, followed by a couple updates/fixes to the jailbreak program the same day.

</comment><comment><USER45> Mirrored the Windows version on MultiUpload. It's the only one I could snag before everything imploded.

<http://www.multiupload.nl/QZTXLPXVHB>

</comment><comment><USER46> Almost all the Cydia sources become slow because the heavy load.

</comment><comment><USER47> As a developer the jailbreak is conflicting.

On one hand it's good to get all the cheap skates off ios 5.1 but then alternatively were going to see a bigger piracy spike now.

<comment><USER48> That's an awfully big assumption that only pirates don't update to the latest and greatest device. Especially when many can't afford to and a smaller community does actually jailbreak to mod their devices. Some I know that went back to iOS from Android were waiting for a jailbreak so they could mod stuff.

<comment><USER47> It's not an assumption, it's a fact. Our stats show that 84% of the pirated copies of our apps in the last 30 days users have been on 5.x.

</comment></comment></comment><comment><USER49> Does anybody know what the state of unlocking A4+ chips is after this release? Stuck with a friggin Bell Canada iPhone 4S I can do shit all with

</comment><comment><USER50> I installed Sms GV extension, then the iPhone 5 keep rebooting... I recovery it try again, them keep rebooting again.... ><

</comment><comment><USER51> Already Jailbroken. Thank you Evasi0n!

</comment><comment><USER17> Can't connect for 30 minutes now, stuck on the "Just a moment..." screen.

Can anyone provide mirrors?

<comment><USER52> Here's a dropbox link for the windows version. It'll be up until there is too much activity: [https://dl.dropbox.com/u/6469722/evasi0n-win-1.0-3c53ba10e24...](https://dl.dropbox.com/u/6469722/evasi0n-win-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.zip)

</comment></comment><comment><USER53> I am very afraid of this moment. I know after this jailbreak released, there will be more and more people use illegal applications, and I have to worry about the meals next month. For god sake, hoping they will not release something that could break the IAP.....

<comment><USER54> Your concern is silly. People pirating applications are likely not going to be willing to pay for those applications in the first place. There is no opportunity cost associated with piracy.

If anything, in my opinion, app piracy just increases the visibility of your app and could potentially win new, legitimate users through word of mouth.

<comment><USER55> Citation needed.

Do you really think that software pirates with jailbroken iPhones that have their phones loaded up with the latest popular apps would not spring the 20 bucks or so in the app store to buy them if it were hard to pirate them? The "no opportunity cost associated with piracy" canard held water in the days where lots of commercial software was reasonably outside the reach of your average individual who wanted to play with it. But saying that little Johnny that's playing Angry Birds for free would not have bought the game for a dollar anyway is, to be frank, complete bullshit.

<comment><USER56> Not the OP, but the only people I've heard of pirating apps are those for whom $20 is the difference between eating and starving (e.g. students whose phone is paid for by family, but their food and apps are not).

<comment><USER16> A number of people pirating apps are people who don't have the App Store available in their countries, and some others are children and young teenagers who don't have their own money to buy things online (with parents who aren't interested in helping them buy games).

</comment></comment></comment></comment></comment><comment><USER57> Warning: jailbreaking your phone is now illegal:

[http://investorplace.com/2013/01/jailbreaking-your-phone- wil...](http://investorplace.com/2013/01/jailbreaking-your-phone-will-be- illegal-after-jan-26/)

<comment><USER58> I always wonder, how can you U.S. guys tolerate this stupid carrier lock-in with year-long contracts? When in other parts of the world they are planning or already have laws obliging carriers to provide call forwarding to your new number after you switch away.

<comment><USER6> It's happening in Europe too, and I don't see the big problem.

It's what makes it economically possible for carriers to offer a subsidized up-front price on the phone. You're free to buy the much more expensive unlocked phone, even in the US, on apple.com, no?

<comment><USER59> Its completely unnecessary. If you sign up for a 1 year contract, you still have to pay out the rest of the contract, regardless of whether your phone is unlocked or not. The only reasons to lock the phone are to force users to pay roaming charges, and to stop people from switching carriers _after_ the contract has expired.

</comment></comment></comment></comment><comment><USER60> Jailbreak was smooth as can be on my iPhone 4S Sprint.

</comment><comment><USER61> Does it really work? ever for New iPad (3gen iPad)?

<comment><USER62> I just jailbroke my iPad 3 with it, works like a charm, on iOS 6.1

<comment><USER61> omg. can't wait get to my ipad to do this. christmas once again.

</comment></comment></comment><comment><USER63> JB worked like a charm! Thx evasi0n! :-)

<comment><USER63> On my iPhone 5.

</comment></comment><comment><USER64> Has Cydia keeled over for anyone else?

<comment><USER35> Yup.

</comment></comment><comment><USER65> can someone provide a mac download link?

<comment><USER17> [http://asiekierka.pl/evasi0n-mac-1.0-3c53ba10e2448d311b0f415...](http://asiekierka.pl/evasi0n-mac-1.0-3c53ba10e2448d311b0f4157f2d7eb568f106c4f-release.dmg)

[https://mega.co.nz/#!5h0BwQoa!KdRLFwNJ3OjMS-7Zs2YGQnsvPxAKEs...](https://mega.co.nz/#!5h0BwQoa!KdRLFwNJ3OjMS-7Zs2YGQnsvPxAKEsaAjabY__8pNtY)</comment></comment>


So you want to write your own CSV code (2014) - <AUTHOR> http://thomasburette.com/blog/2014/05/25/so-you-want-to-write-your-own-CSV-code/ <comment><USER2> I wrote my own CSV code. That's only about 10 issues, and I understand them all, and they're easy to handle, and to test (unlike, say, threading issues, or OS bugs that don't happen on my computer).

That wasn't even close to being one of the more difficult parts of my program to write. I've had to do zero work to maintain it. I don't seem to have ever had any bugs filed against it. Are these supposed to be difficult? Sure, don't write your own encryption algorithm, and avoid touching threads if at all possible, but CSV?

Sure, I wish I could have used a library. Unfortunately, once you start adding requirements, the options start to vanish. It needs to be a C (or C-callable) library. It needs to accept input as Unicode strings (lines), not just files on disk, or bytes. It needs to return results line-at-a-time. It needs to have a compatible license. It needs to support the same package manager as I'm using (or be a trivial to include, like a single file). It needs to be reliable (essentially bug-free, or at the very least currently maintained). It needs to be fairly efficient. What library supports all these?

> Writing CSV code that works with files out there in the real world is a > difficult task. The rabbit hole goes deep. Ruby CSV library is 2321 lines.

I love Ruby, but that file looks a little nuts to me right now. The block comment at the top of csv.rb is over 500 lines long. The Ruby module tries to do everything. Just "def open(filename, mode="r", __options) " and its docstring are 100 lines long! This class includes methods for auto-detecting the encoding, and auto-detecting common date formats, and some kind of system "for building Unix-like filters for CSV data".

I don't put all of that into my CSV parser, because I want my code to be modular. I have features like data detectors and filters and open-file-by- filename in my program, too, but they're fully independent, not tied to any one file format. When I come up with a better way to detect dates in text, I don't want to tweak a 6-line regex in every file format I've written.

<comment><USER3> I wish the JVM was less of a pariah. I love being able to just reach for [https://commons.apache.org/proper/commons-csv/user- guide.htm...](https://commons.apache.org/proper/commons-csv/user-guide.html)

Here is the entire logic of the CSV parser (test code is in a separate path) [https://gitbox.apache.org/repos/asf?p=commons- csv.git;a=tree...](https://gitbox.apache.org/repos/asf?p=commons- csv.git;a=tree;f=src/main/java/org/apache/commons/csv;h=83af61a0d67b70f708125b0ca9d41b09a9396178;hb=refs/heads/master)

<comment><USER2> I'm not as anti-JVM as I once was, though the recent licensing changes have left me confused and nervous. I read their licensing FAQ (again) and I still can't tell if I'm allowed to include recent versions of their JVM in my application.

Were I to use the JVM for my background processing needs (the thought has occurred to me!), I'd definitely use Clojure -- 'data.csv' is less than 150 lines, including comments, for both the parser and formatter!

<comment><USER3> I suggest looking into very specifically AdoptOpenJDK.

[https://adoptopenjdk.net/faq.html](https://adoptopenjdk.net/faq.html)

A lot of people have been digging into the general insanity of Java licensing since Java 8's divergence. Here is a good overview [https://medium.com/@javachampions/java-is-still- free-2-0-0-6...](https://medium.com/@javachampions/java-is-still- free-2-0-0-6b9aa8d6d244)

</comment></comment></comment></comment><comment><USER4> One thing that has always puzzled me is why CSV exists, given the ASCII unit/record seperators appear to do exactly what the comma and newline characters are attempting to do with codes which could appear within records, and have been well defined since some point in the 60s.

And here we are, 60 years later still struggling to work out where a record ends...

<comment><USER5> Easy: CSVs are human readable and writeable and humans tend to have a comma sign on their keyboard, while they don't have the ASCII record seperator visible (teaching them to use eg Alt + 30 will not work).

And if they had that key on their keyboard then you'd have the comma problems all over again: What if a ASCII record seperator shows up in the field?

<comment><USER6> But humans rarely use notepad these days. They use code editors like Coda or VSCode at the very least which have all kinds of advanced features. Surely, those can include support for ASCII separators?

<comment><USER7> Let's say we have a product master. Something looks fishy and we want a brand person or product supply person to check... Push a CSV and it's trivially easy for them to open in XL and check, make corrections and send.

In a business context, this happens far more often than you may expect. Sure you can build a custom platform to validate and make people connect to a form connecting to a database - but Excel is a great user interface and __everyone __knows excel.

A funny incident - we were struggling to build a complex workflow in KNIME where at some points we need user input. Nothing out of the box was great - tools either assume a dashboard paradigm or a data flow paradigm - nothing In between.

One of our creative folks came up with the solution of writing to a CSV and getting KNIME to open it in excel. The user would make changes and save, close excel and continue the workflow in KNIME. Even completely non technical people got it.

<comment><USER8> We had to ban our non-technical users from opening CSVs in Excel because it would strip leading zeros and do other exciting things like convert large numbers to 1.23123E+20 :(

It does this without any notification or warning when you save.

<comment><USER9> That behavior is technically correct because those are all valid representations of that value, e.g., 007449938 = 7449938 = 7.449938E6. Pretty much any plain text (or binary) data format will have this same issue, unless it supports additional metadata for display purposes.

If you intended the data to be a string value, then it should have been enclosed in quotes.

<comment><USER10> The problem is that when you have a number like 123451234512345, it may get rewritten as 1.234512E+14, and then you are loosing information.

</comment></comment></comment></comment></comment></comment></comment><comment><USER11> This scary stuff only applies if you need to _read_ garbage CSVs from a million different sources. Even then, like ken says here, that's only like 10 issues to figure out.

Exporting is likely the more common case and is pretty simple. Quote fields if they contain \r\n",

I'd say relax, focus on your particular problem - if a simple 2-3 line export loop fits your use case and is simpler than a csv library (e.g. in C/C++ perhaps where dependencies are a pain) then why not.

This fear mongering about "you can't possibly do this, the library writers are much smarter than you" can lead to it's own ridiculousness of hundreds of dependencies and ungodly waste of processor time, memory, build time, containerization hassle, etc. to solve what should be a simple problem.

<comment><USER12> I agree. Sometimes the business case is needing to read .CSV files that have some pretty bizarre properties and you need a massive library that takes care of all of that.

In my case (I do a LOT of .CSV work) though none of these problems exist. I haven't reached for Python's .CSV library in years and neither have my co- workers. We simply loop through the file, split the strings on commas, and have a few if statements to parse on a situation by situation basis. Extremely old school, but it works very well and is easier than dealing with the objects that are generated from using the library. I realize this probably doesn't work with all use cases on HN though, or even most.

My ideal language has direct support for making file I/O, dictionaries, sets, whatever all in core and easy to use. This keeps me from having to write my own helper classes and modules or bring in 3rd party libraries. Other languages like to have a community package for everything to where you have 9 options for everything. This seems to be common in JS and Perl and is certainly valid, but I'm not a big fan. Python hits a sweet spot for me.

</comment></comment><comment><USER13> The Pandas CSV reader (Python) is one of the best and most versatile implementations out there. Also very fast.

[https://pandas.pydata.org/pandas- docs/stable/reference/api/p...](https://pandas.pydata.org/pandas- docs/stable/reference/api/pandas.DataFrame.to_csv.html)

<comment><USER14> Disagree.

Data.table's fread is leagues ahead of pandas.

[https://www.rdocumentation.org/packages/data.table/versions/...](https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fread)

Fread has automatic footer detection and has automatic skip logic to help parse out mangled headers in some csvs.

<comment><USER15> fread can also accept a shell command which it will run and use the result as input. I use that to grep the lines I want from files too big for RAM.

<comment><USER14> Same here!

I also use fwrite to send it back to the shell to continue my processing pipeline.

[https://www.rdocumentation.org/packages/data.table/versions/...](https://www.rdocumentation.org/packages/data.table/versions/1.12.8/topics/fwrite)

It's mind boggling how little fanfare data.table has despite it being the best way to handle tabular data in 2020.

</comment></comment></comment></comment><comment><USER16> Oh .csv, how I love to loath thee.

I have a funny story about .csv. Back in the day I was working on an integration with Fishbowl. Looking through their docs, the data format was your typical XML type stuff, until you got to the import requests. And there, lo and behold, was XML wrapping, you guessed it, .csv. I literally laughed out loud.

[https://www.fishbowlinventory.com/wiki/Fishbowl_Legacy_API#I...](https://www.fishbowlinventory.com/wiki/Fishbowl_Legacy_API#Import)

Years later they released an updated API. Upon checking it I discovered it's now _JSON_ wrapping .csv, so, you know, progress.

[https://www.fishbowlinventory.com/wiki/Fishbowl_API#Import](https://www.fishbowlinventory.com/wiki/Fishbowl_API#Import)

<comment><USER17> Paths in SVG are another example where the designers couldn't bring themselves to do _everything_ strictly in XML and decided to use (space) delimited values:

       <path d="M 100 100 L 300 100 L 200 300 z"
            fill="red" stroke="blue" stroke-width="3" />

<comment><USER18> This is sort of a quasi standard way of describing paths as drawing operations. You'll find this in working with fonts as well and old plotters sort of.

</comment></comment></comment><comment><USER19> > Do you really still want to roll your own code to handle CSV?

No one really does, but we get backed into it due to these various issues.

The general issue is that there is no real standard for csv, yet it's often used as the touch-point for integration of disparate systems!

The integration goes something like this:

1\. A and B decide to integrate! A will accept data from B.

2\. A and B want to use a good interchange format, but after weeks of intense negotiations B can only get her IT department to agree to deliver csv (and feels lucky to get that much).

3\. B uses excel to provide sample and test files, which are used by A for development or even in early stages of the live integration. Things seem to be going smoothly.

4\. At some the export process at B changes, a new csv generator is used and now things break on A's side. Or something on the data input side changes at B so that new forms of data are now present that their csv generator does handle the way A expects, etc.

<comment><USER20> Eh—I had to do it recently. There was a file format that I was dealing with that was basically a combination of CSV for data and Yaml for headers (and some other edge case stuff) and the Python CSV reader couldn't handle some edge case that we needed handling, mostly around reporting back to the file provider where they messed up one thing or another in the file.

It didn't really take me that long. Lots of easy to write tests and the thing is processing a ton of data now for a well known stock exchange. Far easier than, say, the machine learning stuff I've written.

</comment></comment><comment><USER21> Funny story on this subject. When I was transitioning from academia to industry, I was afraid (having browsed hackernews so much) that I would be harshly judged for reinventing the wheel. So I removed the CSV library I had written from Github. Shortly thereafter, I was emailed by a well-known programmer (the creator of a fairly major language) asking why I had removed my CSV library, which he was using in one of his projects!

Here's the library, btw: [https://github.com/semitrivial/csv_parser](https://github.com/semitrivial/csv_parser)

</comment><comment><USER22> I think the most fun I had parsing a CSV file was when I found out it switched from UTF8 to UTF16 after several thousand lines, as differently encoded CSV files had been concatenated together directly into a single file...

<comment><USER23> I have a similar story: parsing a CSV file that had an unexpected 0x1A byte in the middle. Turns out the file was concatenated together in multiple passes from multiple files using the COPY DOS command, which by default adds ASCII SUB to the end as an EOF marker: [https://en.wikipedia.org/wiki/Substitute_character](https://en.wikipedia.org/wiki/Substitute_character)

<comment><USER24> I have done this to myself in the past. Those were....dark days.

<comment><USER23> The sad thing is that this happened last year.

<comment><USER25> You don't say, someone sent a CSV like that to me yesterday...

</comment></comment></comment></comment></comment><comment><USER26> Well, considerations like these are typical for more or less any kind of software you will ever write. Because of this all but the very simplest software needs automated tests.

But one may also note that your particular usage of CSV may be a bit simpler than supporting all of the possible complications. Especially if one is not reading CSVs that could come from anywhere.

It depends, having dependencies is always a risk and a nuisance as well. The library you are using may have bugs, acquire bugs in the future, change in incompatible ways and so on.

</comment><comment><USER27> At a former job we processed files, many CSV, from hundreds of different sources and I was in charge of cleaning up the code that did this once. There were a few tiny binaries without source called csv2tsv and csv2tsv2. No documentation of any kind of course.

csv2tsv just handled quoted fields and I was able to replace it with a few lines of Python without issue.

The csv2tsv2 program was used for CSV files from exactly one company. We couldn't ask them for technical assistance - it's likely their systems had been written years ago and running without updates since then - so I tried to figure out what the binary was doing. The input file had some null characters in it, but they seemed to be used inconsistently, and I was never able to figure out what that binary was supposed to do.

I left that binary alone and kept using it for a few years before a new guy joined and took over the system from me. I mentioned this weird old binary to him and over the course of a week he poked at it now and again before figuring out what it was doing. He used to work at a bank and realized it was using the same quoting method that some old data format they'd used there did - something about doubling characters and a few other tricks.

There's nothing simple someone won't make complicated.

<comment><USER24> You are thinking of these: [https://www.w3.org/Tools/csvtotab- vv](https://www.w3.org/Tools/csvtotab-vv)

Agreed, they are excellent.

</comment></comment><comment><USER28> CSV is awesome :-). Leon Bambrick curates an Awesome CSV Page [1] and another Awesome CSV Page [2] by my humble self.

[1]: [https://github.com/secretGeek/AwesomeCSV](https://github.com/secretGeek/AwesomeCSV) [2]: [https://github.com/csvspecs/awesome- csv](https://github.com/csvspecs/awesome-csv)

<comment><USER29> > When the final cockroach breathes her last breath, her dying act will be to > scratch her date of death in a CSV file for posterity.

CSV is the Keith Richards of file formats.

</comment></comment><comment><USER30> I can relate. I have been one of those who started writing a CSV library and went down the rabbit hole of edge cases and configurability. Looking back (and only thinking on productivity) my time would have been better spent doing something else. I did, however, learn quite a lot along the way:

\- how text is represented in a computer (and all its caveats),

\- how to identify and tune slow code,

\- how to vectorize code (thanks to Daniel Lemire's [https://www.youtube.com/watch?v=wlvKAT7SZIQ](https://www.youtube.com/watch?v=wlvKAT7SZIQ) ),

\- how to encapsulate code better,

\- how to use some great CSV tools out there (thanks to Leon's [https://github.com/secretGeek/awesomeCSV](https://github.com/secretGeek/awesomeCSV) )

\- how to better manage and communicate an open source project.

All in all, I am still not sure it was a mistake. Moreover when Swift didn't really had, at the time, a good Codable interface to CSV. Next time I encounter a similar problem I will probably just wrap a C library and create a good interface, though.

In any case, if you are coding in Swift and find yourself in need of a CSV encoder/decoder, you might want to check [https://github.com/dehesa/CodableCSV](https://github.com/dehesa/CodableCSV)

</comment><comment><USER31> My initial comment was going to be that "The Python CSV library is only 448 lines (378 LOC) [1]".

Only later did I realize that the bulk of the work is done by the C library, which is ~1500 LOC [2].

I guess parsing CSVs reliably is a reasonably difficult undertaking.

[1] [https://github.com/python/cpython/blob/3.8/Lib/csv.py](https://github.com/python/cpython/blob/3.8/Lib/csv.py) [2] [https://github.com/python/cpython/blob/4a21e57fe55076c77b0ee...](https://github.com/python/cpython/blob/4a21e57fe55076c77b0ee454e1994ca544d09dc0/Modules/_csv.c)

<comment><USER29> I have a base "library" I've used for several years that wraps the csv module (specifically DictReader/Writer) to handle encoding issues as well as data parsing. Have run into minor issues, but think that module alone is one of Python's best use-cases.

</comment></comment><comment><USER32> And this is just the tip of the iceberg for data importation.

Managers and customer facing folks want to say _Can 't you just use off the shelf software for this?_ If your data importation needs are at all non- trivial, the problems tend to come in 2 categories.

1) Error reporting from off the shelf software is inadequate. If you are lucky, it will consistently point you to the correct row number. It's usually lacking any additional info.

2) If your application is at all non-trivial, you have specific importation needs, not easily covered by 3rd party generic solutions. For instance you want to import well formed tables with primary keys. Lots of solutions claim to handle this case, but you usually run into problems under category (1).

A programmer can overcome these difficulties and muddle through 3rd party solutions, but you probably don't want to hand this over to users. It just won't be a good experience for them.

I completed an importation library tailored for our specific needs and handling the real problems we have experience over many years. The front-end project to make this available to users started and stopped and changed hands over months. We are finally nearing deployment, and so far the users we have shown it to like it.

</comment><comment><USER33> An additional problem is that Excel, the tool many people use to open CSV files, uses the computers regional settings for decimal separator and thousands separator. CSV files are used as an interchange format, so the format of the numbers in it does not depend on the region of that specific computer.

You can tell Excel to always use '.' as a decimal separator, but then it also uses it for presenting numbers to the user. It boggles my mind that software like Excel doesn't understand the difference between reading/formatting numbers for human use and reading/formatting numbers primarily meant for talking to other software.

<comment><USER34> Excel will also take a number with a leading zero and drop it while it converts it from csv, for ex 01234 -> 1234, which is SUPER bad if you are dealing with any sort of IDs to invoices or work orders from outside the company

<comment><USER35> That is a good occasion to curse Excel (how does it know the number isn't octal, too?) and also curse the person or group that designed a textual coding system consisting of digits with leading zeros. Looking at you, National Drug Code.

</comment></comment></comment><comment><USER36> Well sure, if you're writing a generic CSV parser, it's very complicated. But if you know what your input looks like, CSV can be one of the simplest format to parse.

</comment><comment><USER37> I think you need to be careful with choosing to use CSV format.

CSV works fine as long as you are not handling character stings. Character strings get messy as soon as they might introduce commas or characters outside of ASCII.

For tabular data which has character strings I prefer to used the ODF format [http://opendocumentformat.org/developers/](http://opendocumentformat.org/developers/) (.ods file extension) which has good import and export capabilities from Excel, or Google sheets. If the user needs the data in CSV for entry into another application then they can convert within Excel and handle any conversion problems themselves.

<comment><USER38> Do you have a preferred library for python for example)?

Debian packages python3-odf and [https://github.com/eea/odfpy](https://github.com/eea/odfpy) seems to still have some activity (vs the other listed).

<comment><USER37> I don't. In the past I have just used an XML and a zip library like this article explains [https://www.codeguru.com/csharp/csharp/cs_data/xml/article.p...](https://www.codeguru.com/csharp/csharp/cs_data/xml/article.php/c16311/How- to-Read-and-Write-ODFODS-Files-OpenDocument-Spreadsheets.htm)

</comment></comment></comment><comment><USER39> I am the lead developer behind the Node.js CSV parser [0]. The project started in 2010, an hour before me leaving work for the weekend. It ended up being a long, very long journey that isn't over yet. The project grew incorporating many features I would never have expected, fixing corner cases the several hundred tests were not covering and teaching me a lot about coding and project handling. The bottom line, writing a correct CSV parser takes a little bit more than implementing an advanced string spliting function.

[0] [https://csv.js.org/](https://csv.js.org/)

</comment><comment><USER40> There's this system, which has become extremely popular for processing data (Apache Spark). And its default configuration cannot parse CSVs properly and can lead to pretty severe data loss. I wrote about this a while ago and it's still an issue. (Yes, we did lose data his way, it's not a theoretical exercise.)

[https://kokes.github.io/blog/2019/07/09/losing-data- apache-s...](https://kokes.github.io/blog/2019/07/09/losing-data-apache- spark.html#csvs-dont-have-a-standard-per-se-but-lets-be-reasonable)

</comment><comment><USER41> I wrote my own CSV code rarher thsn using a library because the input CSVs were produced by guys who didn't use a library. The columns had a fixed "real width" but also had a column that contained an indeterminate number of comma- separated, values. One can escape commas in CSV for just such a purpose, but these guys didn't. So, to deal with this issue quickly, I had to basically parse the line forwards until I hit the variable-length field, then backwards until I hit it as well. Then I could parse the field to determine how many values it had and extract them.

</comment><comment><USER42> My CSV story from last night:

I had these 15MB+ excel sheets and was trying to open them with Apache POI. I gave that code a generous 2GB memory: GC overhead limit reached.

Then I opened them in Excel, saved them as CSV, reached for a CSV-library and was done within seconds.

Ok, well, the story is more about how parsing XML can ruin your day ;)

<comment><USER43> Are you using POI's eventmodel APIs to handle such files? The SAX model isn't too bad for reading files.

</comment></comment><comment><USER44> It was not emphasized in the article, but the quotes can be both single and double.

That said, it's futile to try to write universal CSV parser. If you need to write your own, just make it possible to choose delimiter and type of quotes and call it a day. LibreOffice Calc does same.

</comment><comment><USER45> I coded a few robots.txt parsers i.e.: (these are just the ones i cared to publish) [https://github.com/franzenzenhofer/simple-functional- robots-...](https://github.com/franzenzenhofer/simple-functional-robots-txt- parser) [https://github.com/franzenzenhofer/robotstxt](https://github.com/franzenzenhofer/robotstxt)

and even though robots.txt seems like a very, very simple text based protocoll there are unanswered mysteries

the biggest mystery is user agent groups and comments

i.e.:

      User-agent: googlebot
      User-agent: bing
      User-agent: yandex
      Disallow: /

so i am disallowing everything for google, bing, yandex; easy enough, but:

      User-agent: googlebot

      User-agent: yandex
      Disallow: /

means that googlebot has no instructions, but yandex is disallowed all.

if a whole line is commented out

      User-agent: googlebot
      #User-agent: bing
      User-agent: yandex
      Disallow: /

is a commented out line a blank line or a non existing line?

if it is interpreted as a blank linke, the disallow only counts for yandex, if the line is non existant, it counts for googlebot and yandex.

i like simple things, but sometimes the complexity is between the lines.

<comment><USER46>
      User-agent: googlebot
      #User-agent: bing
      User-agent: yandex
      Disallow: /

why would it be interpreted as a blank line? If you remove everything after the #, that includes the new line characters at the end of the line. Leaving:

      User-agent: googlebot
      User-agent: yandex
      Disallow: /

<comment><USER47> It's ambiguous, and this is why significant whitespace can be so frustrating. Unless it's specified, some people will interpret a full-line comment as a blank line with a comment ending it (`^#. _$`), and others will interpret it as you have (`^#._?\n`). Neither is obviously correct (even if it's obvious _to you_ ).

Edit: I don't know how to escape in HN formatting. Obviously there are italics where literal asterisks should be.

<comment><USER48>
      *** You can just use three asterisks. ***

__* You can just use three asterisks. __*

       Unfortunately you need something after them though. ***

Unfortunately you need something after them though. ___

<comment><USER47> Thanks, that's good to know!

</comment></comment></comment></comment></comment><comment><USER25> > What if there is non ASCII data?

> Just use utf8 right? But wait…

> What if the program reading the CSV use an encoding depending on the locale?

> A program can't magically know what encoding a file is using. Some will use > an encoding depending on the locale of the machine.

Excel (for Mac at least) is a fucking pain in this regard. Just try this minimal UTF-8 example:

      tmpfile=$(mktemp /tmp/XXXXX.csv); echo '","' > $tmpfile; open -a 'Microsoft Excel' $tmpfile

Hooray, you successfully opened

      ","

I'm not even sure in which encoding e280 9c2c e280 9d corresponds to that (not the usual suspect cp1252, nor any code page in the cp1250 to cp1258 range; easy to confirm with iconv).

One remedy is to add a BOM (U+FEFF) to the beginning of the file, but of course no one other than Microsoft (at least in my experience) uses this weird UTF-8 with BOM encoding (which the Unicode standard recommends against), so it breaks other programs correctly decoding UTF-8.

This means I can never share a non-ASCII CSV file with non-technical people. Always have to convert to .xlsx although it's usually easier for me to generate CSV. Then .xlsx opens me up to formatting problems, like phone numbers being treated as natural numbers and automatically displayed in scientific notation... Which means ssconvert or other naive conversion tools aren't enough, I need to use a library like xlsxwriter.

I'm not sure why it's so hard to just fucking ask when you don't know which encoding to use. (Plus it's not super hard to detect UTF-8. uchardet works just fine. Plus my locale is en_US.UTF-8, maybe that's a hint.)

<comment><USER49> > Hooray, you successfully opened

> ","

> I'm not even sure in which encoding e280 9c2c e280 9d corresponds to that

That'll be MacRoman (not very surprisingly).

[https://en.wikipedia.org/wiki/Mac_OS_Roman](https://en.wikipedia.org/wiki/Mac_OS_Roman)

<comment><USER25> Ah, I thought it has to be something Microsoft.

We can place the last modernization effort to this piece of code, then.

</comment></comment></comment><comment><USER50> Implementing your own CSV parser is one of the first lessons a green software engineer will learn in how to NOT try to re-invent the wheel.

Still, please don't import packages with just one goddamned 3-line function in it.

<comment><USER51> It's what happens when DRY becomes dogma instead of a guideline...

This leads to debacles like the npm left-pad & kik affairs and other scary shit like [https://github.com/parro-it/awesome-micro-npm- packages](https://github.com/parro-it/awesome-micro-npm-packages)

Yeah - have fun maintaining dependencies when every other module depends on one-liners that can be pulled or broken at random...

</comment></comment><comment><USER52> One small tip if you receive a CSV with a non-default separator (like ;):

* Open it in a text exitor

* Add "sep=;" on the first line (without the quotes)

* Now at least Excel will open it as intended

<comment><USER53> IIRC Excel will ask for the separator anyway when you open the file?

<comment><USER54> It does not and actually assumes locale-dependent separators (usually semicolons if your locale uses a comma as a decimal separator and comma otherwise).

If you want to specify separators you have to go via Text Import.

</comment></comment></comment><comment><USER24> To be fair, sometimes calling some giant java library or installing tons of dependencies is just not realistic. Use the library for the mission critical stuff or the automated reporting without a doubt, but it's important to be practical and thinking on your feet as well.

In the meantime for quick analysis and testing 90% + can be accomplished with one line of (g)awk.

      awk -v FPAT='"[^"]*"|[^,]*' -v OFS='\t' '{$1=$1; print $0}' "$filename"

More on FPAT [here]([https://www.gnu.org/software/gawk/manual/gawk.html#Splitting...](https://www.gnu.org/software/gawk/manual/gawk.html#Splitting- By-Content))

<comment><USER55> You can also use xsv for this kind of analysis of CSV files. Which is a unix- style tool designed for CSVs and with a proper CSV parser.

It's written in Rust so it's one binary - no runtime dependencies, and will happily chunk through multi-gigabyte files.

[https://github.com/BurntSushi/xsv](https://github.com/BurntSushi/xsv)

</comment></comment><comment><USER56> Reminds me of [https://infiniteundo.com/post/25326999628/falsehoods- program...](https://infiniteundo.com/post/25326999628/falsehoods-programmers- believe-about-time)

<comment><USER57> It applies to everything, dates, floating point numbers, strings, etc. What we would consider the basics always turn out to be much deeper than they appear on the surface.

<comment><USER58> Dating advice: if you habitually, unironically, and without good cause use any dating format but that which is mandated by ISO 8601 we cannot be friends, let alone more.

[https://twitter.com/jwiechers/status/1205515440543424513](https://twitter.com/jwiechers/status/1205515440543424513) \+ the thread

</comment></comment></comment><comment><USER59> After working int he world of CAD tools for decades, I cannot tell you the number of times I've seen a younger engineer try to write a CSV parser only to fall into a circular hacking deathspiral.

You can't just use regex/split to handle CSV, unless you have significant field cleaning BEFORE converting to csv.

In reality you need lexical analysis and grammatical rules to parse any string of symbols. This is often always overlooked by naive implementations.

I take issue with OP's claim that RFC4180 is not well-defined, but almost all of the cases the OP listed are literally in the spec.

</comment><comment><USER60> Whilst we're talking about libraries, had good experience with CsvHelper for C#/.NET.

[https://joshclose.github.io/CsvHelper/](https://joshclose.github.io/CsvHelper/)

Documentation isn't 100% but it's a good tool. Been using it to work on a production 15+ column SQL->CSV mapping job with tens of thousands of records, working great.

</comment><comment><USER61> I wrote a CSV-reading library 16 years ago that I'm pleasantly surprised handled most of the points of trickiness the article mentions.

[https://www.neilvandyke.org/racket/csv- reading/](https://www.neilvandyke.org/racket/csv-reading/)

The documentation section "Reader Specs" gives a good idea of the different variations it intends to handle. I was working from a sense of variations I'd seen myself, and extrapolating.

One thing I also did, which I didn't see in the article, was to support comments.

Unfortunately, at the time I wrote it, there was no standard Unicode support for Scheme, so I didn't get into that, but just used the character abstraction (which might or might not involve parsing of a particular non-ASCII encoding).

Fortunately, AFAIK, it's worked for people, since fixing/extending it at this point would mean relearning the code. :)

</comment><comment><USER62> Those are the simple things to worry about, it can even get more complicated in the real world. So complicated that writing our own parser was the only way to go.

So what are some of the more advanced challenges:

* Banking systems that treat CSV export just as some kind of line based dump file with no regard of consistent formatting. If the banking backend was updated, the format might change within a file from one line to the next

* Some misconfigured data dumping pipeline parses CSV the wrong way from another system and emits it in escaped form again in a different format. For instance putting a complete line escaped into a single field. Your parser has to detect that there is a CSV embedded within a CSV.

* Dumping Pipeline treats \r\n as all kinds of silly lines and re-embedding those lines with quotes around them to look like real data

* Inventing completely new specs of what CSV could mean

* Use mainframe character-sets from the last millennium

</comment><comment><USER63> > Some countries use a comma as decimal separator instead of a colon.

And some companies... like Klarna, fixes this wrongly. At least they did a few years back.

We'd get a CSV file from them, containing payment information, but they ran into the issue that they really wanted to use comma for field separation, but also for krone/øre (decimal) separation. They ingenious solution: Fields are now separated by ", " that's a comma and a space.

Pretty much no CSV parser will accept a multi character field separator. So many fields would just have a space prepended to whatever value was in that field and you'd get a new field, when the parser split the field containing the amount. So now you have say 5 headers but each row would have 6 fields, because the money amount became two fields.

Being on the receiving end you now have to choose, do you want to strip spaces and reconstruct the amount field manually, or parse the CSV file yourself.

<comment><USER64> If you use .NET then CsvHelper[0] supports multi character separators.

[0] [https://joshclose.github.io/CsvHelper/](https://joshclose.github.io/CsvHelper/)

</comment></comment><comment><USER65> I totally agree with this. Writing your own csv code is the archetypal example, in computer programming, of reinventing the flat tire.

I've found that every major language has at least one reusable component / module / library / dll to to this. They're maintained by people who have lots more patience than I do.

</comment><comment><USER28> FYI: I've put together a collection of CSV specifications [1]. CSV Dialects / Flavors include CSV v1.0, CSV v1.1, CSV Strict, CSV <3 Numerics, CSV<3 JSON, CSV <3 YAML and some more.

[1]: [https://github.com/csvspecs](https://github.com/csvspecs)

</comment><comment><USER66> Fantastic article!

It's missing: \- "what if the quotes are smart quotes?" \- This applies to both quotes inside a field and quotes wrapping a field \- "what if the file doesn't have a header?" \- "what if a logical record is spread over multiple lines?" \- Yeah... so, you have line 23 that has the majority of the data in the expected columns, line 24 has additional data in columns 23, 25, and 28 that's supposed to be appended onto the previous line's data that also appears in columns 23, 25, and 28. Line 25, same thing, but line 26 starts a new logical record.

Seen all three above in the wild and Python's (pretty awesome) CSV code didn't handle them. Queue the custom code that wraps Python's core CSV parsing.

</comment><comment><USER67> Would've been a lot nicer if they used the record and unit separator instead of commas and newlines. There are characters made specifically for storing tables like this and they instead chose to reuse characters which might appear in regular text.

</comment><comment><USER68> If you can't write a correct (according to RFC 4180) CSV parser in 15 lines of Perl then you're not a Real Programmer!

(Yes, I understand that not all CSV files in the wild are correct according to RFC 4180. Yes, I also understand that only grey-beard loons still use Perl.)

The strangest thing about CSV, I think, is that if there's only one field per record and the file ends with a CRLF then you can't tell whether there's a final record containing an empty field following that CRLF. It's probably best to assume there isn't.

<comment><USER69> 15? damn, that's 10 lines less than my awk csv parser. oh well...

</comment></comment><comment><USER70> I prefer to use a SQLite database. And if a customer insists on CSV then I still use a SQLite database to import and/or export the CSV without them realising it. SQLite brings sanity to CSVs.

<comment><USER71> You still have to import the CSV though, so the problems are still there, you just don't have to deal with them after a successful import.

<comment><USER70> You misunderstand how enterprise integrations work. It's easy to tell a customer: "your CSV is not compatible with standard off the shelf tooling, indeed the most popular database engine in the world" than "your CSV doesn't seem to work with our software/library/<proprietary ETL framework>". The former makes it their problem, the latter will almost undoubtably make it your problem. In case you don't know: SQLite has a really well designed and very performant CSV import function. BTW did you really downvote me for that? Wow.

</comment></comment></comment><comment><USER72> Just today I needed to write a .csv file in Python on Windows and got a file with CR CR LF line endings by default (\r\r\n). That was using the standard 'import csv' in Python 3.

<comment><USER73> yeah you gotta pass

      with open(...,newline='') as f:

<comment><USER29> I think the docs now include this.

<comment><USER73> yes, they have for some time now i think. I wish it was better known, though.

</comment></comment></comment></comment><comment><USER74> Discussed at the time: [https://news.ycombinator.com/item?id=7796268](https://news.ycombinator.com/item?id=7796268)

</comment><comment><USER75> Sometimes the best thing you do is learn at least one parsing framework so that you can write your own parsers. The advantage of this is you know exactly what the behavior will be.

Here's a CSV parser in ~40 lines of C++ using Boost Spirit that handles all kinds of weird cases:

[https://gcc.godbolt.org/z/KGUHXo](https://gcc.godbolt.org/z/KGUHXo)

It's easy to customize, declarative, and compiles down to about ~15KB.

</comment><comment><USER76> I wrote my own CSV parser for [https://www.easydatatransform.com](https://www.easydatatransform.com). I ran into some cases where it wasn't clear what to do. For example where you had some whitespace outside of quotes. So I checked what Excel and Numbers (the Mac spreadsheet) do. They parse the same string differently...

</comment><comment><USER77> doesn't seem insanely difficult

dont be scared of coding guys, just differentiate between environments: do it for your 4fun project, not prod.

<comment> yoz-y The problem isn't really that it's difficult to solve. It's because it's impossible. There is no real formal spec. The RFC does not specify edge cases and nobody respects it anyways. "Generic" CSV parser is a pipe dream, don't try to do it.

Now, if you define your own set of rules on what is and is not allowed then it's fine.

<comment><USER78> Indeed it's a perfect example to illustrate the difference and the importance of an under-defined spec and a well defined one.

Whatever you don't define, will be undefined. But you still somehow have to do something definite without a definition.

The customer expects predictable output at the end, and until we have not only clairvoyance, but clairvoyance that can be built in a machine, you can't have predictable output without either predictable input, or a spec that actually provides an answer for any input.

</comment></comment></comment><comment><USER79> Even when using libraries to read CSV data the need for sanitization isn't uncommon. Sure there's RFC4180, but variations are common. Mysterious records that are out of sync are typical.

For my own sanity I like to verify that columns are of the right data type where possible.

That said, I still prefer CSV over heavier formats (XML,JSON) where the data conforms.

</comment><comment><USER80> Can we somehow start to use the ascii 30 character aka the Record Separator (SP)? This would solve a lot of the problems!

<comment><USER81> Not as fun as you might think. I had to parse and write inkjet control files for commercial printing. It used record separators, group separators, and unit separators.

The problem? Records began WITH the RS character and there were NO line breaks.

Ended up writing my own library [https://github.com/Wombatpm/VIPNT](https://github.com/Wombatpm/VIPNT)

</comment></comment><comment><USER17> A good one I saw recently was a text file that, even though it had a .csv file extension was actually using another character as a delimiter (in this case colons) but as it contained large amounts of textual data actually had a enough commas to be parsed as a CSV file.

VS Code and a suitable extension handled this rather better than Excel.

<comment><USER82> Sounds like CSV in this case stood for Colon Separate Data

<comment><USER83> Further generalization: Character Separated Values

</comment></comment></comment><comment><USER84> I write CSV code all the time in different languages. It's a fun challenge that incorporates several nuances. It has been a form of code kata for me over the last 20 years. Biggest lesson, one does not simply split a string. You have to scan through and tokenize following the state of a quoted field.

</comment><comment><USER85> ndjson(new line delimited json) is what we should use as a human friendly and machine friendly alternative to csv.

json is a universal format. Works in every major language. It's spec is a state machine defining what to do with every possible encountered byte. [https://www.json.org/json-en.html](https://www.json.org/json-en.html)

SIMDJSON parses json at RAM speeds (~3Ghz).

CSV should die. It's a terrible format. Just send arrays of ndjson instead. You'll probably have a few more " and [, but hey, you can deeply nest arrays inside arrays of objects with arrays.

[https://github.com/simdjson/simdjson](https://github.com/simdjson/simdjson)

<comment><USER86> JSON doesn't handle int64 or explicit date-time data types, which makes it nearly as painful as CSV in practice when used for data integration. Both of those are explicit types present in nearly every real-world database schema.

So you end up with many of the same problems as CSV. You basically get an application-specific JSON dialect to interpret.

Header row or not? Is a long series of digits in quotes an int64? What happens when someone puts an ISO-formatted date string into a field that is supposed to be a simple string?

</comment></comment><comment><USER87> Some fast implementations in C++?

In particular, I tried in the last weeks to work on a OpenMP / MPI driven CSV parser, but surely accounting for new lines inside quotes can be a pain the ass, I'm wondering if there's a go-to implementation or model to implement that.

</comment><comment><USER88> I did this just for fun to learn a parsing library I was using. It didn't deal with utf-8 though, but even that is manageable. The complexity with csv is that it is a family of formats with small changes. But if you make your parser configurable, you are set.

<comment><USER88> If you want to do something really hard, try to parse PDF, that is sheer insanity.

</comment></comment><comment><USER89> CSV is not a portable format across applications. That's it. You should use it only in your own application or in connection with applications that you know. The big problem is to think that you can handle a generic CSV file, because there is no such thing.

</comment><comment><USER90> Just because all kinds of inputs exist, doesn't mean you should support handling them all.

</comment><comment><USER91> What about implementing something that works for your environment and making it fail-fast so that any unexpected input will just abort+log the current operation? You could then expand on that on-demand.

<comment><USER78> This assumes you can recognize unexpected input.

<comment><USER91> I assume unexpected input results in either an error (or exception, whatever your language calls it), a crash or a user complaint. I also assume it's unlikely for an input parser to introduce a security issue into your app if it's written in a safe language and your app has proper validation set up (i.e. don't trust input as soon as it passes the parser).

</comment></comment></comment><comment><USER92> I don't want to but there is nothing that I've found in c# other than a helper tool that will only translate objects to CSV rows.

</comment><comment><USER93> I'm currently implementing a REST API that returns CSV files.

</comment><comment><USER94> Me too. Everything was great until Mr O'Connor came along.

</comment><comment><USER95> >Some countries use a comma as decimal separator instead of a colon.

I kind of wish we could all just pick _something_ and stick with it.

<comment><USER96> That sentence must have a typo in it, right?

Wikipedia [1] doesn't list _any_ country that _does_ use the colon (:) as the decimal separator, so possibly the period/dot (.) as used in most English- speaking countries, was meant?

1: [https://en.wikipedia.org/wiki/Decimal_separator](https://en.wikipedia.org/wiki/Decimal_separator)

<comment> yoz-y Yeah, I think what author meant "some countries don't use the period but a comma, thus hilarity ensues" (especially when your code respects the locale and the csv file is written in append mode)

</comment></comment></comment><comment><USER97> i wanted to... but not anymore :/

</comment><comment><USER98> import csv

</comment><comment> GuB-42 I disagree with the article, I'd say writing your own CSV code is the second best option, behind not using CSV at all.

The reasons are:

\- CSV is really simple. A complete RFC4180 parser shouldn't take more than 100 lines of readable code to implement. Even less so for the writer. So much that most of the code is likely to be interface code (file I/O, matching your internal data structures, etc...) rather than parsing. And chances are that a library won't even make your code shorter.

\- As mentioned in the article, CSV is not well defined. I've seen all sorts of weirdness: semicolons vs colons, LF vs CRLF, simple vs double quotes, backslash escaping, comments, floating point numbers with comas instead of points. What your library understands as a CSV may not be the files you have to work with. I'd rather have a simple piece of code that correspond to what I am given rather than a big library that uses AI techniques to try to guess the intent of whatever wrote this file.

There are cases where it would be foolish not to use a library. I will never write a XML parser for instance, it is a well defined and complex format. JSON is borderline but I still won't write my own parser. CSV is essentially a proprietary format, so it is fitting to use a proprietary parser/writer.</comment>


Weird neutrino excess won't go away, hints at new physics - <AUTHOR> https://arstechnica.com/science/2018/06/weird-neutrino-excess-wont-go-away-hints-at-new-physics/ <comment><USER2> A few other comment threads about this:

[https://news.ycombinator.com/item?id=17210982](https://news.ycombinator.com/item?id=17210982)

[https://news.ycombinator.com/item?id=17225957](https://news.ycombinator.com/item?id=17225957)

[https://news.ycombinator.com/item?id=17253127](https://news.ycombinator.com/item?id=17253127)

<comment><USER3> This one has new info, though:

 _Now Fermilab is back with its latest update, using two additional years of MiniBooNE data. The excess is still there, and it has edged even closer to the statistical standards for discovery. If you combine the Fermi and Los Alamos data, we 're already there. It's looking more and more like another break in the Standard Model, and the possible explanations include an entirely new type of neutrino._

<comment><USER4> Given the publication dates, those other articles are almost certainly also based on [https://arxiv.org/abs/1805.12028](https://arxiv.org/abs/1805.12028), which incorporates those two additional years of data.

</comment></comment></comment><comment><USER5> > a proposed additional type of neutrino, called a sterile neutrino

Does this imply the existence of a new lepton?

<comment><USER6> Do you mean a new generation of matter, with two new leptons and two new quarks?

Cosmological data show that there should be only three generations of matter.

<comment><USER5> > Do you mean a new generation of matter, with two new leptons and two new > quarks?

Yeah, that's what I was getting at: one of the leptons being the sterile neutrino and the other being the "sterile particle".

</comment></comment></comment><comment><USER7> Is there a guide to science for people like me who have barely grasped Special Relativity. I am a full _century_ out of date and I do wonder if I should be worried before we throw the stuff I don't know out and replace it with other stuff I don't know

<comment><USER8> Watch PBS Space Time channel starting from the early videos:

[https://www.youtube.com/channel/UC7_gcs09iThXybpVgjHZ_7g/vid...](https://www.youtube.com/channel/UC7_gcs09iThXybpVgjHZ_7g/videos?view=0&sort=da&flow=grid)

They range from easy to follow to quite technical.

</comment></comment><comment><USER9> Dumb question that I feel I may have asked before, but that I don't recall the answer to:

Why does it seem like everyone believes the number of "fundamental" particles is finite? What fundamental reason is there for this?

<comment><USER10> Because there is no evidence of an infinite number of different fundamental particles.

<comment><USER9> What would count as evidence for this?

<comment><USER10> Particles have energy levels. There is a maximum energy level and a minimum energy level. We already know that every energy level does not have a particle associated with it (quanta). Therefore there cannot be an infinite number of particles.

<comment><USER9> Huh, interesting. What is the maximum energy level a particle can theoretically have? How do we know it is the maximum?

<comment><USER10> [https://en.m.wikipedia.org/wiki/Absolute_hot](https://en.m.wikipedia.org/wiki/Absolute_hot)

Tl;dr, put enough energy in a particle and it turns into a singularity.

</comment></comment></comment></comment></comment></comment><comment><USER11> Wow. 1,959 detections out of 10^21 interactions. And they talk about statistical significance. In any other field that is so close to zero you would ignore the results.

<comment><USER12> Think of it this way: With 10^21 samples, you can be pretty certain that your sample mean will be the theoretical one.

<comment><USER4> The absolute error will be low, but the relative error can still be high. E.g. if you're dealing with events that have a frequency of 1 in 2 million, then 1 million samples can't have exactly 0.5 occurrences, so the relative error is at least 100%.</comment></comment></comment>


Ask YC: what is your most important idea? - <AUTHOR>

Two hundred years from now, what piece of work would you like to be remembered for most?<p>If you had one year to devote to working on one project, after which you could never code again, what would you make? <comment><USER2> Lao-Tzu would argue that being of lasting significance, is, well, insignificant:

       Of the best leaders
       When the task is accomplished
       The people will remark
       We have done it ourselves.

The people who made TCP/IP and HTTP are among the greatest benefactors to humankind in the past 20 years. Yet hardly anyone knows who they are, and I doubt they will make the history books. Because they were so successful, their achievement now seems like an impersonal product of the age. If only!

All the internet entrepreneurs, all the ones who became richer than Croesus - their achievement was to become gatekeepers for a zillionth of the wealth that the Internet and the Web have bestowed on the world. And then they became targets for other people, who had piled up maybe a zillionth and a half of that wealth, and people like Arrington can't shut up about them.

If you want to change the world, and the people know your name, you're doing it wrong.

<comment><USER3> I think, to be remembered for long time it's not about the field or occupation with which a personality is associated but it depends on the quantum of fundamental shift that he/she brings in the evolution of civilizations. eg: Newton and Einstein will be remembered for the centuries to come though other brilliant physicists will not. Humanity will respect Wright brothers for many ages to come, though the case may not be same for the person who invented jet engine.

May be their original work is not in much use these days but they opened a new domains for future generations to work on and that created the shift.

<comment><USER4> Or -- just playing devil's advocate here -- perhaps it's a matter of PR... e.g., Edison vs. Tesla.

</comment></comment></comment><comment><USER5> An open collaborative platform for education. Something along the lines of OpenCourseWare. I feel like that should be the future if it isn't already in sight. I'm of the school of thought that puts educating people as a top concern, and the more readily available and collaborative it is, the better it can be for everyone.

<comment><USER6> Selling things to .edu institutions is a royal pain in the neck. Couldn't tell if this is non-profit pie-in-the-sky idea or for-profit venture, but if it is the latter, best of luck :)

<comment><USER7> maybe there are applications for enhanced training within companies, or even to sell on the cheap to college students themselves that want a better group- study environment for when they can't meet up. you don't have to sell bulk licenses.

</comment></comment></comment><comment><USER8> 200 years from now noone will remember anything you'll ever accomplish. Look back to 1808, do you honestly remember any major inventions from back then?

I don't remember the exact quote, but it goes something along the lines of "If you want to be remembered 100 years from now, you need to kill a lot of people"

<comment><USER9> Really? I don't know about 1808 exactly, but history is full of accomplished individuals whose names we remember many years later. Galileo? Gutenberg? Edison? Ben Franklin? Marie Curie?

<comment><USER10> But Hitler, Genghis Khan are always more famous than Edison or Galileo. Just like how G.W.Bush or Dick Cheney will be more famous than Sir.Tim Berners Lee for the future generation.

<comment><USER11> Yeah, but Steve Jobs and Bill Gates will be more famous than Bush or Cheney.

<comment><USER8> How many people remember Rockefeller and Carnegie off the top of their head? Actually bad example, it hasn't been 200 years since those guys.

How about Vanderbilt and Astor? I doubt you'll find many people who know who they are.

Wealth alone doesn't get you into history books, you may fund some expedition, but it'll be Louis & Clarke who'll get into the history books.

<comment><USER11> Not wealth, but ideas. Steve Jobs and Bill Gates are not just icons of wealth, but icons of revolution, in a way.

I don't think Warren Buffet will be remembered, which I find sad. But I believe Jobs will live on as a sort of legend of his own.

<comment><USER12> Weren't Rockefeller, Carnegie, and Vanderbilt "icons of revolution"? I mean, oil, steel, and shipping are more important than ipods, right?

<comment><USER11> Not to the masses they weren't. But let's see who remembers Henry Ford nowadays. He's the icon of that era.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> The same idea I'm working on this year. Something that makes it easy to do a task that's simple offline, hard online. (I'd say more but I'm a paranoid youth.)

Really, anything that does the generic idea presented above. Whenever you present a simple method of doing something, the whole system tips.

</comment><comment><USER13> Showing that P=NP.

<comment><USER14> Just show that N=1 or P=0. Problem solved. Next?

<comment><USER14> It was a joke. Apparently people here have no sense of humor.

<comment><USER13> I got the joke, not sure why you got voted down.

In any case, I've already gotten the trivial solutions =)

</comment></comment></comment></comment><comment><USER15> My email sig:

"If there's one thing I am truly proud of, it's that I've wasted all my talents, contributed nothing to the human race, and will not be remembered by generations yet to come."

If you're doing things because you want to be remembered for them, you're doing things for all the wrong reasons and will never be remembered for anything.

</comment><comment><USER16> I would take one year with my daughter and travel with her around the world.

</comment><comment><USER17> Making sending money as easy as sending email.

<comment><USER18> Paypal?

<comment><USER19> Paypal is easy. But it is not as easy and ubiquitous as sending an email.

</comment></comment></comment><comment><USER20> Linear time protein folding.</comment>


major labels: the problem with music - <AUTHOR> http://www.arancidamoeba.com/mrr/problemwithmusic.html <comment><USER2> Steve Albini is the man. A great producer and musician, and his analysis here is dead on.

</comment><comment><USER3> A lot like the VC game in many ways.</comment>


Ask HN: How to politely quit my job? - <AUTHOR>

I humbly request the wisdom of fellow HNers,
Folks, I currently work on a very small company with a really great boss (who's a family friend). This company is quite small, we're only 4 persons including my boss and his wife. I am tasked with a really boring (maybe nightmarish) job where I am the single programmer/maintainer of a lovecraftnian designed online app,  this jobs is slowly driving me crazy because I LIKE to focus on quality and the programmers before me focused on fast, so I've inherited a codebase with no docs, a scm with not a single commit comment and a mysql db with 10k tables. In summary, the ambient is great, people are great, job is not.
Now, I've received and accepted my dream job proposal working for a previous contractor on projects I really enjoy but I don't quite know how to bring these news to my current boss. How to approach and tell him that I am leaving in some days. He is a great guy and his family and mine goes way back. I want to talk as soon as possible because I want to help to find a decent replacement and train the guy a little on how to navigate that stuff. So, anyone here ever faced similar situation? WWHND (What Would HNers Do)
Cheers <comment><USER2> It's easy:

You: Boss, we need to talk.

Boss: Ok, let's talk.

You: I decided to leave ${CompanyName}.

Boss: Ok, is there anything we can do to make you stay?

You: No, I've made my decision. How long do you want me to stay to transfer my responsibilities?

Narrator's note: the standard is 2 weeks.

Other things: they might ask you why you're leaving. It's not the time to provide them with a list of complaints (especially if you haven't raised them in the past). You can be vague but truthful: the other opportunity looks like a better fit for you.

That is essentially the script I followed when leaving my past jobs. All my bosses were professional about it and understanding.

</comment><comment><USER3> I'm preparing to put in my 2 weeks notice this upcoming week, too. I'm expecting a job offer that will require me to move out to LA, from TX, on Monday. My current job is similar to yours, minus the family friend aspect.

I am one of 2 developers at the 7 person company. The company is full of enjoyable people, for the most part. The job itself is one that I've wanted to quit for the past year and a half. Perhaps worse in my scenario is that when I leave, they're going to be pretty much left floating dead in the waters until they find another developer. And, even when they do find a replacement, it'll take a few months for that person to get up to speed. I've been there a year and a half and I'm not even 100% knowledgeable about everything there. So, I'm a little nervous about when I put in my 2 weeks notice.

To answer your questions, though - I'm just going to do it. I'm the kind of person that usually stresses out a ton, and would probably be closing to passing out due to stress within the moment prior to putting in my 2 weeks. It's all going to be fine, though. It's not up to my boss, it's up to me. It's my future. So, with that in mind, I'm just going to notify him as soon as possible and that'll be that. I'm going to be polite and thankful for the job, since it has been very accommodating. (they let me finish up school part time while working)

In your case, I'd just be extra friendly. If he's a family friend, then he will understand.

Good luck.

</comment><comment><USER4> I just went through something very similar. I had been working with a smaller company all through school and for a while after I graduated. The people I worked for were family friends and just great to have as employers. Then, I had the opportunity to join another company that I felt would allow me to do more of what I enjoy and had more room for me to grow, so I went for it. It was quite difficult telling my boss that I was leaving, no doubt about it. He offered everything he could to get me to stay, but in the end I knew that it was time for me to move on to something new.

That being said, here's a few tips that I think could help you.

1\. Remember that this is business. That doesn't mean forgetting about the people involved, just that the most important thing in this situation is your career.

2\. Do as much as you can to help the transition. The new company that I went to work for wanted me to start asap, but I was willing to give my previous employer 2 full weeks of my time since I did value the relationship and wanted to leave on the best terms possible.

3\. Just be honest. While they may not want you to go, they should understand that this is what you need to do.

Good luck! Hope it all goes well.

</comment><comment><USER5> I just went through this process with my last employer 3 weeks ago. I just finished week one at the new employer and it is great. Here is what I did.

Gave 2 weeks notice, I told my boss this opportunity was too good to pass up. It was something I really wanted to do.

Offered to do whatever was needed over the next two weeks to help with the transition. (Mostly documentation for the new guy and closing outstanding projects.)

Recommended that they expand a current contract they have with a service provider to help fill some of my roles while they look for a new employee.

I kept the exit interview and discussions with other employees professional and did not complain about the problems of the current job. Really no reason at this point to make a big stink on the way out, I don't like to burn brides un-necessarily.

I hope this helps.

Good luck with the new job!

</comment><comment><USER6> A few disconnected thoughts:

* They own the business, not you. If you owned a part then you would be able to make decisions like spending resources on improving the codebase. In the end, it is up to them to run it, not you.

* Businesses come and go, relationships stay. If you think the guy is a good boss stay in touch; even if your notice causes a lot of bad emotions and someone throws a drama fit, just treat it like a temporary thing and it will be -- call them back in two months to see how things are going and they won't even mention it.

* Don't believe that you owe them any sort of extended stay. You can help their new hire transition via email support from your new place. If you agree to stay longer than 2 weeks, they are likely to delay the new hire by that much.

</comment><comment><USER7> One thing I've found useful to do is to offer to help them find a replacement. This might means asking your friends, but also helping them with interviews (up to a point anyway...)

</comment><comment><AUTHOR> Folks,

This is the OP. Thanks for all the kind words and advise. Yesterday, I talked with my boss and it went surprisingly well (maybe I was pessimist), he understood why I was leaving and I will stay there for two weeks helping the transition.

Again HN community shows how great it is.

Thanks folks!

</comment><comment><USER8> I would try to be polite and as emotionally neutral as possible. People don't always react the way we think they will but, at the same time, if you broadcast your negative expectations too much, some people will react to that even if they would otherwise not have been all reactionary. There is a scene in "While you were sleeping" where the character played by Bill Pullman finally tells his dad that he doesn't want to inherit/takeover the family business. He has been avoiding it for years, while pursuing his real dream on the side. His dad is like "If you had told me sooner, I could have sold it to your uncle". It's very, very anticlimactic, involving none of the emotionalism or drama that Bill Pullman's character feared.

Good luck with this.

</comment><comment><USER9> Did your boss offer stock options?</comment>


Show HN : Org-Mode parsing in Javascript (project stalled) - <AUTHOR> https://github.com/subtenante/orgjs/

<comment><AUTHOR> Hi HN.

I worked on this project last year and have stopped for 6 months now. I'd like some input from the community to know if the project is interesting to others than me.

I was in the process of rewriting the rendering part to make it closer to a CSS approach (having selectors to apply a given template to the matching nodes), but that was a bit too difficult for me at the time, so I hit the pause button to go meditate. The attempts to use Jison for the selector language appear in the 'cleanup' branch, folder 'etc'. Not very convincing but I'm still trying to find the best way to do that part.</comment>