ZFS on Linux: Unlistable and disappearing files - <AUTHOR> https://github.com/zfsonlinux/zfs/issues/7401 <comment><USER2> We are working on it. We know what patch introduced the regression and 0.7.8 is going out soon to revert it. Until then, users should downgrade to 0.7.6 if they have not already. The Gentoo and EPEL maintainers have pulled the affected releases from the repositories (technically masked on Gentoo). Ubuntu was never affected.

The regression makes it so that creating a new file could fail with ENOSPC after which files created in that directory could become orphaned. Existing files seem okay, but I have yet to confirm that myself and I cannot speak for what others know. It is incredibly difficult to reproduce on systems running coreutils 8.23 or later. So far, reports have only come from people using coreutils 8.22 or older. The directory size actually gets incremented for each orphaned file, which makes it wrong after orphan files happen.

We will likely have some way to recover the orphaned files (like ext4's lost+found) and fix the directory sizes in the very near future. Snapshots of the damaged datasets are problematic though. Until we have a subcommand to fix it (not including the snapshots, which we would have to list), the damage can be removed from a system that has it either by rolling back to a snapshot before it happened or creating a new dataset with 0.7.6 (or another release other than 0.7.7), moving everything to the new dataset and destroying the old. That will restore things to pristine condition.

It should also be possible to check for pools that are affected, but I have yet to finish my analysis to be certain that no false negatives occur when checking, so I will avoid saying how for now.

<comment><USER3> > _We will likely have some way to recover the orphaned files (like ext4's > lost+found) and fix the directory sizes in the very near future._

How should people behave right now?

Will normal usage of production filesystems erase data, or will read/write activity leave the potentially-orphaned files in place?

You've also mentioned snapshots being tricky in the thread. Should people stop creating snapshots in case orphaned files are not included in the snapshots?

\--

> _It is incredibly difficult to reproduce on systems running coreutils 8.23 > or later._

IIUC:

\- This is specifically due to the fact that `cp` in 8.23 is optimized (8.22 created files in {0..2000} order, 8.23+ randomized the order (I don't quite understand why))

\- The script in [https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d...](https://gist.github.com/trisk/9966159914d9d5cd5772e44885112d30) uses `touch` to create files in random order and some people reported this triggered the bug

<comment><USER4> > 8.23+ randomized the order (I don't quite understand why)

It's using inode order, which speeds up things significantly on some filesystems:

[https://lists.gnu.org/archive/html/bug- gnulib/2014-02/msg000...](https://lists.gnu.org/archive/html/bug- gnulib/2014-02/msg00007.html)

<comment><USER3> Ah, thanks! I knew there was a rationale but hadn't quite gotten it.

Nice.

</comment></comment></comment></comment><comment><USER5> Wow that's an old Kernel! [https://linux.slashdot.org/story/16/01/31/0424221/linux- kern...](https://linux.slashdot.org/story/16/01/31/0424221/linux- kernel-2632-lts-reaches-end-of-life-in-february-2016)

<comment><USER6> My thoughts exactly, while the RHEL kernel has lots of backported patches - it's by no means complete or near current.

<comment><USER7> Of course, it also isn't intended to be.

</comment></comment></comment><comment><USER8> Probably obvious, but hooray for open source software! What a fantastic response to the bug.

<comment><USER9> Imagine if this had happened to Apple, everyone and his dog would be rolling over each other villifying them.

<comment><USER10> I paid good money to Apple so something like that would not happen, so yeah I'd be villifying them. You bet!

<comment><USER9> As opposed to the free RedHat Enterprise Linux operating system

<comment><USER10> No, I love to hate redhat because they are the most incompetent company in the computer industry's history; just take a look at their priority 1 bugs at bugzilla.redat.com and the picture becomes crystal clear. (I'm forced to perform system engineering on RHEL every day and a lot, so my hate for it grows daily.)

</comment></comment></comment></comment></comment><comment><USER11> Is this ZOL tip or Linux specific? FWIW, the bug does not seem to reproduce on FreeBSD-current (r332158).

However, it also does not reproduce on Ubuntu 4.4.0-116-generic running the ZFS stuff from Ubuntu.

<comment><USER2> Ubuntu was never affected. The regression started in 0.7.7 and Ubuntu is on 0.7.5. HEAD was affected until earlier today when the patch was reverted.

I am not sure if the bad patch was ported to the other OpenZFS platforms.

<comment><USER12> Debian Stable(jessie) is on 0.6.5.9 and Debian unstable(sid) is on 0.7.6 so they shouldn't be affected.

<comment><USER13> Debian stable is stretch, released in june 2017. It's on 0.7.5 currently.

<comment><USER13> correction, I hadn't updated my server in a while, stretch itself is on 0.6.5.9 also, but stretch-backports is on 0.7.6 (not sure when that happened exactly)

<comment><USER14> > but stretch-backports is on 0.7.6 (not sure when that happened exactly)

[2018-03-09] Accepted zfs-linux 0.7.6-1~bpo9+1 (source amd64 all) into stretch-backports (Aron Xu)

source: [https://tracker.debian.org/pkg/zfs- linux](https://tracker.debian.org/pkg/zfs-linux)

</comment></comment></comment></comment></comment></comment><comment><USER15> Looks like they're on top of the causes and solution. But there's a caveat: possible orphans.

[https://github.com/zfsonlinux/zfs/issues/7401#issuecomment-3...](https://github.com/zfsonlinux/zfs/issues/7401#issuecomment-379895691)

<comment><USER2> We will work out a solution for people affected so that they can get those orphaned files back.

</comment></comment><comment><USER16> Regression introduced by [https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce0...](https://github.com/zfsonlinux/zfs/commit/cc63068e95ee725cce03b1b7ce50179825a6cda5), will be reverted and a new version tagged.

</comment><comment><USER17> From title first thought was - "Wow, this is awesome security/privacy feature of the file system!" and then from reading comments I realise it is a bug

</comment><comment><USER18> Great to hear the ZoL guys are right on this. Bravo.

It also reminds me why my NAS runs Debian..

<comment><USER19> It also reminds me why my NAS runs FreeBSD.

<comment><USER2> The bad patch passed review by developer(s) from other platforms. Matthew Ahrens was a reviewer. This was not merged based on unilateral review by the ZFSOnLinux developers. There was also nothing Linux specific about it.

That said, bugs happen. We should be putting new test cases in place to help catch such regressions in the future. If we find more ways to harden the code add against regressions of this nature as we continue our analysis, we will certainly do them too.

<comment><USER20> Reviewers will help catch bugs, but the engineer who writes the code and seeks to integrate it is ultimately responsible for sufficient testing to avoid issues like this one.

<comment><USER2> The author is a fairly new contributor:

[https://github.com/zfsonlinux/zfs/commits?author=sanjeevbage...](https://github.com/zfsonlinux/zfs/commits?author=sanjeevbagewadi)

What do you suggest that we should have done?

<comment><USER20> I'm not sure, but I'm not deeply familiar with this part of the code. In general, I think it's good to be able to induce all of the failure cases for all of the error handling code that's being added.

This can be time-consuming work, but I would argue that this is the file system -- anything less is an unacceptable risk. This quote from my boss comes to mind:

 _Remember: you are (or should be!) always empowered as an engineer to take more time to test your work._ \-- [http://dtrace.org/blogs/bmc/2015/09/03/software- immaculate-f...](http://dtrace.org/blogs/bmc/2015/09/03/software-immaculate- fetid-and-grimy/)

</comment></comment></comment></comment></comment></comment><comment><USER21> i stick with old good ext4

</comment><comment><USER22> So this is not a "ZFS bug", but a "ZFS on Linux" bug. The actual ZFS on systems which have had ZFS for decades is not affected at all.

<comment><USER23> > The actual ZFS on systems which have had ZFS for decades is not affected at > all.

ZFS hasn't existed "for decades".

<comment><USER10> It has existed since 2002, and was officially put back into onnv on 31.10.2005, and made available in Solaris 10 update 2 at the end of June 2006. So, it has existed for at least one decade and more.

<comment><USER23> It hasn't existed since 10 april 1998 therefore "decades" is inaccurate. Furthermore, since it wasn't publicly available from 10 april 2003 (who cares how long it was in development) if you round fairly, then its accurate to say "more than a decade" but "decades" is inaccurate. Because "decades" means "more than 1 decade". It denotes "at least 20 years".

That's regardless of the subject we discuss or how good it is etc.

Anyway, the original parent admitted their mistake (thank you) so I'm done with that discussion.

</comment></comment></comment></comment><comment><USER24> At least it's no btrfs. What a disaster that filesystem's been.

<comment><USER25> Is there a stable (loaded word, I know) versioning file system available for Linux?

<comment><USER26> The simple, unqualified answer is no.

BTFS is official Linux and apparently "ok" for simple use cases, but there are no end of reports of failures when non-trivial RAID modes are used, demanding work loads are applied, device replacement is attempted, etc. and there are performance problems under a variety of conditions. There are enough qualifications on the BTFS status page[1] that I, for one, do not consider it 'stable.' ZFS is a thing on Linux, but it's not in the kernel and _when_ it breaks the kernel developers don't officially care, except when they happen to have a foot in both camps. This situation naturally limits the size of the ZFS on Linux user base; you're one of the few if you're doing it and that's not where most production users want to be. LVM can snapshot logical volumes and produce `crash consistent' volumes independent of the type of file system. That's been my go-to solution given no other alternative.

The fact is Linux has trailed far behind its contemporaries in advanced file systems for... 10+ years now? Not terribly flattering.

I suspect the reason is that most production use of Linux occurs in environments that provide many enterprise storage functions independent of the operating system, so there isn't much pressure to, for instance, harden BTRFS until there aren't major deficiencies. I can snapshot/clone/restore/whatever my EBS volumes any time I wish and I can do similar with my private cloud powervault volumes as well. I trust either of these mechanisms far more than _anything_ Linux has ever provided, including LVM.

[1] [https://btrfs.wiki.kernel.org/index.php/Status](https://btrfs.wiki.kernel.org/index.php/Status)

</comment></comment></comment><comment><USER27> Why is the thread title hyphenated? It made me expect there's a utility called "zfs-bug" causing data loss.

<comment><USER28> The OP appears to speak German natively. In German they are much more hyphen happy than English. It's probably the most common error I see among German speakers typing English.

<comment><USER29> This. German merges words together as their modifier mechanism, whereas English just uses word order to indicate modification. Since it's technically _not_ illegal in English to hyphenate to do the same thing, German speakers tend to do this for English.

It's a very hard habit for them to break. :)

<comment><USER28> One could argue that technically nothing's illegal in English, since there's no governing body, just what's in common use.

Kind of like civil vs common law too, now that I think about it :)

</comment></comment></comment></comment><comment><USER30> basically the reason I like my filesystems a few decades old and mature, I would not trust zfs or btrfs with anything critical.

<comment><USER31> I have been running ZFS on macOS (OS X), illumos and Solaris for a good 7 years or so now. A major part of the reason I switched over fully, despite some warts, was that I experienced actual and significant data rot from stuff I was carrying forward under XFS (IRIX), HFS, etc. I don't consider my personal stuff to go back _that_ long, but I still have things from 1993 or so that matter to me. I did a review around 2009ish, and found that a number of old files had at some point or another become corrupted, including ones I know for sure weren't in 2004. I'd been following ZFS somewhat since Sun had demoed it, including a depressing spell after Apple almost moved to it then quit, but it was my own personal actual losses that pushed me to move over.

End to end checksumming and other integrity features just plain should be universal at this point. Should have been a decade ago or more in fact. We have so much incredibly important data now that is digital only and nowhere else, and memory and storage have both become very cheap at the general population level. It's shameful that anyone should still be losing data or experiencing anxiety years or even decades down the line. Integrity, basic levels of security/privacy, and flexible, high integrity replication should all be native level features of any data store system. "Decades old" filesystems just plain absolutely do not cut it, no does anything newer that doesn't include those promises at least as options. Bugs are unfortunate, and I hope this prompts ZoL and associated projects under the OpenZFS umbrella to double check their automated unit and stress tests. Sun rightly made a big deal of that on release. Even so, I wholeheartedly believe that ZFS or the like are far better primitives for a data storage scheme then older FS (or many newer ones for that matter).

<comment><USER32> I am still somewhat suprised there are still not relatively simple filesystems that don't do this. Make it the one feature it does and does properly. Adding enterprise features will just increase development time (and cost) and add bugs. Basically an online archival filesystem, that's how most people use their home computers.

Requirements:

1\. Checksumming on all files

2\. Minimise assumptions of ram correctness

3\. Disc replication (soft raid 1)

Out of scope:

* Snapshotting

* Subvolumes

* Deduplication

As much as I love subvolumes and snapshotting, I feel like CoW probably adds too much complexity to make it (easily) reliable. Honestly I don't know why turning off CoW on BTRFS disables checksumming, so if anyone can shed some light as to why this is, feel free to point out that my requirements are far more complex than I think.

<comment><USER33> Checksumming is like a myth on HN already. No, it's not useful on its own to the user or to the filesystem, but it is useful to the filesystem if it can transparently heal the data. Whole disk replication doesn't address your reliability concerns as much as you think either, doesn't work all that well in general and is a burden to maintain. Doing it properly requires treating the filesystem as a distributed system where disks can join and leave, where everything is sharded, where rebalancing, self-healing, syncing, resyncing is all automatic and blazing fast. And so on.

Nowadays most of the work in storage systems is in distributed systems, and local filesystems are treated as just another unreliable layer.

Hope this gives you some ideas why there are no filesystems like that.

<comment><USER32> Not particularly. I'm thinking for the home user who just copies the photos from their camera to their home computer. Distributed systems would be great, were it not for the fact that by and large, home connections have abysmal upload speeds.</comment></comment></comment></comment></comment>


Meteor 0.5.7 released: major scaling update, new DDP version, EJSON - <AUTHOR> https://www.meteor.com/blog/2013/02/21/meteor-057-major-scaling-update-new-ddp-version-ejson <comment><USER3> I got my first taste of Meteor at a meetup in NYC on Tuesday night. I hacked something together with no experience in about 30 minutes that would be a pain to write in anything else, and completely trivial to write in Meteor. In fact I didn't bother to read the documentation. The app is a shared grocery list: <http://www.teamgrocery.com/>

The app never deletes anything, and I suppose a grocery list shared among all HN users should get overwhelmed really quickly :)

I can't remember the last time I have had so much fun playing with a new framework. Meteor is going to be my goto technology for hackathons/prototypes.

<comment><USER4> Shows an empty list for me and hangs, can't input anything...

</comment></comment><comment><USER5> There's no place except on HN where Meteor gets regular attention, upvotes and always the same fan posts (i.e., "Meteor is going to be my goto technology", "The Meteor guys are moving fast").

Guys, this is all so obvious.

EDIT: downvoting won't help, people aren't that stupid, $11M funding and sponsored posts are no reason to use any framework

<comment><AUTHOR> [meteor dev and OP] Sorry to see this here. I don't know any of the people who commented here so far. I just smiled reading their stories -- they're the reason we work so hard on this project.

If you're ever in SF, come to a Devshop and meet some of the community in person. I think it will change your impression. <http://www.meetup.com/Meteor- SFBay/events/103016662/>

<comment><USER6> If he can't make it to SF, he could also come to a Meteor meetup group in Chicago, Minneapolis, Paris, Toronto, Vancouver, New York, Zurich, Philadelphia, Boston, The Hague, Johannesburg, Portland, LA, London, Austin, or Hong Kong.

[http://meteor.com/blog/2013/02/06/meteor-devshop-0-share- kno...](http://meteor.com/blog/2013/02/06/meteor-devshop-0-share-knowledge- not-germs)

</comment></comment></comment><comment><USER7> As a developer using meteor, most of the changelog is a bunch of technical "jumbo humbo" to me. However, the full release notes said something about "performance" for some day to day operation, and that's all I need to hear.

I must note that meteor does seems to excel at being absurdly trivial to get things done. It also isn't quite of a heavyweight to learn as compared to other frameworks.

(I am working on a time tracking app that's coming rather nicely)

</comment><comment><USER8> We just got Meteor compiling on Pogoapp (using a buildpack which uses meteor's git/master, so incorporates these updates[1]), and I booted up open source demo apps whipped up by a couple London developers after a chat in #meteor IRC:

<https://github.com/alanshaw/meteor-blackboard>

<https://github.com/olizilla/goto-meteor>

here's live demos (you can use zoom in both):

<http://blackboard.pogoapp.com>

<http://meteor-goto.pogoapp.com/>

Be sure to check go take a look at the code, or lack thereof - <USER2> impressive stuff. Since looking just a few months ago, the whole meteor ecosystem seems to have expanded and matured at a pretty incredible clip.

[1]: <https://github.com/oortcloud/heroku-buildpack-meteorite>

</comment><comment><USER9> So can someone tell me what the monetization scheme is for Meteor and other frameworks like it? Is there one? Or is this purely for the joy of the code? Seriously...

<comment><USER6> Meteor CEO here. miles_matthias's got it right.

We think of it like the difference between git and GitHub. git is free and open source, both the client and the server. But many people prefer to use GitHub, which puts a nice management interface on top of the server. And if you are a company and want GitHub inside your firewall, then you can write a check and download GitHub Enterprise, which is a copy of GitHub that comes as a machine image that you start on your own hardware.

Our plan is to build nice management tools for Meteor deployments, and make money primarily by selling an enterprise-ready version of those tools to companies that are large enough to need them, analogous to GitHub Enterprise.

Ultimately the big idea behind the company is to improve the developer experience of the entire application lifecycle, from writing your app, to testing it, to deploying it, to monitoring and scaling it. I think that modern frameworks need a story for all of these pieces.

</comment></comment><comment><USER10> For folks who'd like to try out Meteor in the browser without any local installs, we just wrote a blog post on how you can do so on Action.IO: [http://blog.action.io/2013/02/21/build-meteor-apps-in-the- br...](http://blog.action.io/2013/02/21/build-meteor-apps-in-the-browser-with- actionio-and-mongolab.html)

(Disclosure: I'm one of the guys working on action.io - we're private beta now but sending out invites at a pretty fast clip)

</comment><comment><USER11> The Meteor guys are moving fast. This is awesome. I can't wait until all the current methods of creating web apps are obsolete.

<comment><AUTHOR> [core dev] Four of us are not "guys" :) They move fast too!

<comment> shawn-butler Seriously, we have to moderate the use of the word "guys" now?

It's a colloquialism for "people" in the plural. Although it is kind of interesting that it is common to use "guys" to describe either a group of men, or a mixed-gender group of people but I don't think anyone would use it to refer to a group consisting solely of women.

<comment><USER9> true. further, many languages refer to a group of women in the feminine, a group of men in the masculin and a mixed group in the masculin.

</comment></comment></comment></comment><comment><USER12> Interesting to read such polar views about Meteor.

At Wigwamm, we wanted to build a user experience that was simple. A real-time auction and an auction catalogue full of full screen photographs.

We discussed at length whether Meteor would allow us to do everything we needed. The conclusion was interesting: if it's not simple, if doing through Meteor is too hard, we probably don't want/need to do it.

For us, it's nice to work with technology that genuinely pushes boundaries.

Little plug: we're London UK based, so if you're local, love building products that help people and want to build in pure JavaScript/Meteor, please get in touch (@WigwammHQ on twitter)

</comment><comment><USER13> If Meteorite is the Package Manager for Meteor, why are the two not in sync? E.g. like npm for node?</comment>


Everything is Fashion - <AUTHOR> http://stianj.com/fashion/ <comment><USER2> Getting strong Vaporwave vibes here

</comment><comment><USER3> Yes, that fashion for destroying infectious diseases using vaccinations, drugs, and hygiene is certainly fleeting, isn't it?

</comment><comment><USER4> This is awesome. I love it.</comment>


3 wheeled, $10K EV, Sondors accpeting reservations - <AUTHOR> https://sondorselectriccar.com/ <comment><AUTHOR> I found this quite interesting, and here are some links if anyone wants to go through:

\- Quick car facts, scroll down [https://sondorselectriccar.com/about/](https://sondorselectriccar.com/about/)

\- Invest/reserve with $100 [https://www.startengine.com/sondorscar](https://www.startengine.com/sondorscar)

\- Decent review from Roadshow [https://www.youtube.com/watch?v=jZnifrNXLig](https://www.youtube.com/watch?v=jZnifrNXLig)

\- Another review from Engadget [https://www.youtube.com/watch?v=kCHzu- aD-9I](https://www.youtube.com/watch?v=kCHzu-aD-9I)

</comment><comment><USER2> This is the first time I've actually seen equity crowdfunding. I'm used to companies talking about "investors" when they really mean donors or preorders.

I've been thinking about the three wheel issue lately. In CA, a three wheeled vehicle is a motorcycle, so it does not need to meet automobile safety regulations, which means it can be made much lighter and therefore more efficient.</comment>


Foucault in California - <AUTHOR> https://www.the-tls.co.uk/articles/public/michel-foucault-lsd-death-valley/ <comment><USER2> I always found it strange that Foucault has come to be identified, in the mainstream view of his work but also in academic circles, with radical leftism. It's hard to read Discipline and punish or any of his work on biopolitics and not sense the underlying criticism of the welfare state and "big government". His libertarianism made him at odds with many French Marxist thinkers of his era and I suppose must have played a part in his relocation to America.

I think that Foucault found capitalism and the freedom it granted absolutely fascinating (just like Marx before him was fascinated with capitalism's ability to increase wealth), which eventually led him to look into thinkers like Hayek, Friedman and Say. Anedoctal but imo significant: his last partner and the executor of his will now works for a neoliberal think tank.

The left-wing magazine Jacobin had ran an interesting inyerview on the subject of Foucault's neoliberal inclinations some time back, worth a read if the subject interests you: [https://www.jacobinmag.com/2014/12/foucault- interview/](https://www.jacobinmag.com/2014/12/foucault-interview/)

<comment><USER3> That's interesting, I wasn't aware that he had turned rather libertarian. But I must ask, did he agree with everything in standard libertarianism, or did he have some significant differences?

<comment><USER4> > standard libertarianism

What's that?

<comment><USER3> You are apparently aware of significant differences among libertarians, so why don't you list some.

<comment><USER4> That's a baffling reply. I think the idea of there being a "standard libertarianism" is quite odd but it's your phrase that could use some clarification, not mine.

</comment></comment></comment></comment></comment><comment><USER5> I'm by no means an expert on Foucault, but I highly recommend learning about some of his ideas or even better, read some of writings. You may not come away agreeing with him (especially right away) but his ideas on freedom deeply impacted me.

You'll see him criticized and reviled in many philosophical circles as being the epitome of post-modernism (which has (unfairly IMHO) become a bit of a dirty word in some cultures), but don't let that scare you. I believe (mostly from reading analysis from people much smarter than me, like Thaddeus Russell) that Foucault is widely misunderstood.

<comment><USER6> The overwhelming majority of people that criticize post-modernism have little- or-no actual knowledge of it. Foucault can indeed be difficult to understand, but make no mistake, he does have interesting things to say.

<comment><USER7> Many who criticize post-modernism (and Foucault) aren't really criticizing the philosophy, but the understanding of that philosophy among groups who have adopted it for political ends.

When people say "post-modernism sucks" they mean, for example, that moral relativism sucks, or that it is stupid to view science as a grand narrative of no more value than other narratives. When they say "Foucault sucks" they are attacking the understanding among the "woke" of how power relations work in society, an understanding that is drawn, however incompetently, from readings of Foucault.

Leading post-modern philosophers didn't embrace those positions in an uncomplicated way, but many influenced by them do, and they massively outnumber the people who have actually bothered to read Derrida or Baudrillard.

Furthermore, many post-modern philosophers were showboaters who enjoyed being provocative, so you don't have to look hard to find justification for the unnuanced understanding of their work.

<comment><USER8> What do you mean by "woke"?

<comment><USER9> I would guess that meaning : [https://www.urbandictionary.com/define.php?term=woke](https://www.urbandictionary.com/define.php?term=woke)

A word currently used to describe "consciousness" and being aware of the truth behind things "the man" doesn't want you to know i.e. classism, racism...

</comment></comment></comment></comment></comment><comment><USER10> Nobody has articulated my feelings about Foucault et al better than Chomsky:

> There are lots of things I don't understand -- say, the latest debates over > whether neutrinos have mass or the way that Fermat's last theorem was > (apparently) proven recently. But from 50 years in this game, I have learned > two things: (1) I can ask friends who work in these areas to explain it to > me at a level that I can understand, and they can do so, without particular > difficulty; (2) if I'm interested, I can proceed to learn more so that I > will come to understand it. Now Derrida, Lacan, Lyotard, Kristeva, etc. --- > even Foucault, whom I knew and liked, and who was somewhat different from > the rest --- write things that I also don't understand, but (1) and (2) > don't hold: no one who says they do understand can explain it to me and I > haven't a clue as to how to proceed to overcome my failures. That leaves one > of two possibilities: (a) some new advance in intellectual life has been > made, perhaps some sudden genetic mutation, which has created a form of > "theory" that is beyond quantum theory, topology, etc., in depth and > profundity; or (b) ... I won't spell it out.

<comment><USER11> That's an excellent observation. Those philosophers are deliberately ambiguous for the sake of profundity. It's not unlike interpreting poetry or literature -- they even built that into their tradition, like a preemptive defence against their lack of clarity. The most honest way to approach Foucault and others like him is to read them as thinkers and artists and to accept what insights you can glean from it and disregard the rest and shake off its overbearing seriousness.

<comment><USER12> Deliberately ambiguous sentences mean 1000 meanings to 1000 followers. Then these 1000 postmodern intellectuals write treatises on the meaning of this, by adding their own ambiguity. In the end, everything goes.

</comment></comment></comment><comment><USER13> An article by Dundas and Wade was briefly discussed in 2017: [https://news.ycombinator.com/item?id=15260411](https://news.ycombinator.com/item?id=15260411)

One detail is that shortly before his death, Foucault sent a letter asking if he could come to live with Wade and Stoneman. "I think he wanted to die like Huxley. I said yes, of course."

<comment><USER4> This looks great and I missed it the first time. Thanks.

</comment></comment><comment><USER14> I find it interesting that many here reading these comments base their professional lives within the STEM fields but yet are so ardent to support the post-modernist philosophers who believe reason to be an absurd meta-physical starting point. It's a little ironic to say the least.

</comment><comment><USER3> Foucault's great focus on sexuality as central to happiness indicates that he never escaped the influence of his Catholic upbringing.

Think of all the things that you care about, that make your life good or bad. Sexuality is obviously one, but only one, and even it is intertwined with many others, like health, child-rearing, companionship, and so on.

But with Catholicism, or at least a rigid version, life is all about fighting sin, and central to sin is sexual desire, and hence fighting sexual desire is central to living properly. Now stick with that sex-centered view of life and just make sexual desire good instead of bad, and you have Foucault, or at least a large part of his views.

You know, someone who follows exactly what his father tells him to be in life is not free. But someone who spends his life doing the exact opposite, and never discovers for himself what a good life would be, which might in some ways agree with his father's vision, is still not really free.

No doubt some Foucault fans will claim, and perhaps properly, that I am greatly distorting his views. But as a public intellectual, Foucault had a responsibility to make his views clear enough that people could understand them and decide if they are right or wrong, and if the latter put them into practice in their lives and society as a whole. It is, for instance, pretty clear what Marx thought should happen in the world, and likewise the American founding fathers.

If it is the case that Foucault wrote in a way that predictably would lead to his being misunderstood, then I say he was being irresponsible. If on the other hand his views are not of any use to most people, then why do they matter?</comment>


Archiving Yahoo Messages: update - <AUTHOR>

Yahoo Messages is going down in a week, and we're trying to archive it before then. https://news.ycombinator.com/item?id=5427558 But Yahoo has very strict rate-limiting which means we need more IP addresses to download from.  Because of the rate limiting, it won't take much of your bandwidth. We just need more IPs :)
There's a pre-built VirtualBox image to run locally http://www.archiveteam.org/index.php?title=ArchiveTeam_Warrior or simple instructions for AWS spot instances https://gist.github.com/duggan/5226732
PS Join us in IRC irc://irc.efnet.org/warrior or http://chat.efnet.org:9090/?channels=%23warrior <comment><USER2> I want to congratulate Archiveteam. You do amazing work in saving dying pieces of the world's knowledge.

Warrior is fantastic software that makes it super easy for anyone to chip in and help. Even on a laptop in VirtualBox, it causes little noticeable battery drain.</comment>


Why Hashbrown Does a Double Lookup - <AUTHOR> https://gankro.github.io/blah/hashbrown-insert/ <comment><USER2> If memory overhead (load factor) isn't a big issue, readers may also find cuckoo hashing rather interesting. (Theoretically, it has a worst-case constant lookup time. Learned about it in one of my university algorithms classes, but have yet to see an implementation in practice.)

[https://en.wikipedia.org/wiki/Cuckoo_hashing](https://en.wikipedia.org/wiki/Cuckoo_hashing)

<comment><USER3> This seems to assume that all information needed to determine if two keys are equal is incorporated into the resulting hash value.

It's premised on the idea that you can dynamically change the hash function such that _eventually_ you will find one with fewer than some fixed number of items sharing the same hash value.

There are two problems: 1) It completely undermines the "worst-case constant lookup", because there is no upper bound on the number of times you might have to change the hash function and rebuild the table before you find one with sufficiently few collisions.

2) It is a much stronger requirement on the keys than other hash tables have. With other hash tables I can simply omit "difficult to hash" data from my hash function, on the basis that _enough_ other data is hashed to avoid a significant performance penalty. With this implementation, the entire hash table will simply fail.

This additional failure mode is also a security concern - if an untrusted user can affect what data is added to the table, it could cause a severe DoS where the table will either enter an infinite loop trying to find a non-conflicting hash function, or have an unexpected failure mode (most people don't expect their hash tables to randomly fail). Even if it is somehow designed such that a hash function can always be found, the attacker could spend time up-front finding values that take a long time for that hash function to be reached, causing numerous re-hashes of the table.

<comment><USER4> What are you talking about?

> It's premised on the idea that you can dynamically change the hash function > such that eventually you will find one with fewer than some fixed number of > items sharing the same hash value.

Why would you think there are any fixed numbers? The OP is about a stdlib hash table (for Rust). Comparable hash table implementations all resize the table when necessary to decrease the load factor. Performance with respect to load factor still matters, of course, less resizing is better, but there are no fixed numbers of anything.

> It completely undermines the "worst-case constant lookup", because there is > no upper bound on the number of times you might have to change the hash > function and rebuild the table before you find one with sufficiently few > collisions.

Why would you need to change the hash function or rebuild anything during lookup? It's lookup. Nobody is changing anything.

> It is a much stronger requirement on the keys than other hash tables have.

What hash table doesn't require keys to be hashable?

</comment></comment></comment><comment><USER5> The author is also in the thread on /r/rust if you have any questions for them: [https://www.reddit.com/r/rust/comments/b38cwz/why_hashbrown_...](https://www.reddit.com/r/rust/comments/b38cwz/why_hashbrown_rusts_new_hashmap_implementation/)

<comment><USER6> I went to university with him. Really nice guy. Every time we talk, I say something like "any day now dude, I'm going to really get into Rust" and then I never do because I'm awful.

<comment><USER7> I've been saying the same since last year. I'm finally picking it up just trying to use it for regular everyday scripting work. It's been slow but atleast I'm starting now.

<comment><USER8> Slow indeed. Too slow...

</comment></comment></comment></comment><comment><USER9> This is pretty amazing, i really like the fact that a fastest hash table implementation is going to be part of the Rust's stdlib.

<comment><USER10> Where does it say that this is the fastest hash table implementation?

<comment><USER11> It's accepted knowledge that the Google Swisstable is currently the fastest hashtable around. (The C++ implementation) But there are three variants, and this looks like the slowest of the three, but the best for dynamic languages.

<comment><USER12> It should depend a lot on what you put into it and the access pattern, no? You can talk about averages, about complexity, about benchmarks, but I feel like "fastest hashtable around" is a very odd expression, as if to desire the world to be a lot less complex than it is and for there to be exactly one best.

<comment><USER11> As I said there are two best, plus the worse third variant. Not implemented by google, only described, but now apparently implemented in Rust. Haven't checked closely, because Rust is kinda unreadable compared to C what exactly is going on. Need to see the SIMD assembly also. You cannot really trust much what's going on with Rust, as there's too much hype and lies, but it undeniably got better recently.

</comment></comment></comment></comment></comment><comment><USER13> So - what's the motivation to use "open addressing" vs chaining, which I thought was the more common approach to solving this.

I assume there must be a substantial performance gain for this to be used as it seems significantly more complicated, any information on how much better it is?

<comment><USER14> All* high performance hashtables use open addressing, because chaining tends to mean (multiple) indirection to addresses outside the table.

* not sure if that's _literally_ true, but I've never seen anyone do chaining in performance-sensitive applications, and all the papers on fast hash tables use some way of open addressing.

<comment><USER13> Interesting, both the Qt and Java ones seem to use chaining, but I guess they're not designed with these sorts of extremely demanding applications in mind.

<comment><USER15> One thing to note is that this performance tradeoff is actually the reverse of what it was ~20 years ago. This is because CPU speeds have improved dramatically more than RAM speeds over a similar period.

Open addressing does do more comparisons than chaining. It used to save time to traverse the linked list rather than to spend CPU cycles on those additional comparisons. Now, because CPU cycles are relatively cheaper, the opposite is true.

The reason the Qt and Java hashtables use chaining is simply that the code in question was initially written back when the tradeoff ran the other way.

<comment><USER16> If hash codes are stored inline alongside keys, code comparisons can be made without an indirection while key comparison (often strings, seldom anything without an indirection) usually needs an indirection. Hash codes are very cheap to compare.

These indirections have always been costly on any machine with virtual memory. TLB misses aren't free. I'd have put any estimate on the tradeoff being the other way around to more than 25 years.

</comment></comment></comment></comment></comment><comment><USER17> Answer: SIMD.

What looks like "two loops" is really "do two simple SIMD operations".

<comment><USER18> I think rather the answer is:

Tombstones in place of deleted elements mean that finding out that key is not in the table requires going to a different place than finding the right place to finally put it. Hence two lookups. SIMD just makes keeping two lookups separate economical.

</comment></comment><comment><USER19> _> And so our "two loops" are usually just "do two simple SIMD operations". Not quite so offensive when you say it like that!_

No, it's not quite so offensive, but this doesn't explain why it's the _best_ option. Is there no equally-fast way to write the first-tombstone implementation with SIMD instructions? The answer seems to be in the sketch of the implementation, which I'm having trouble understanding.

EDIT: I'm watching the original SwissTable talk now... would it really have been worse to use 2 bits for empty/tombstone/full/sentinel, and 6 bits for hash prefix?

EDIT 2: More implementation info. Tombstones are actually rare, because if any element in your 16-wide chunk is empty, you don't have to create a tombstone. In the very best case (a perfectly distributed hash function), your hashmap has to be ~94% full before it's even possible to fail this. Because tombstones are so rare, it's better to save the single bit for extra confidence in the hash prefix.

So, here is my understanding of the implementation and its rationale:

* Every bucket in the backing array has a corresponding byte in a metadata array

* 1 bit of this byte stores whether the bucket is empty, the other 7 bits are for a hash prefix

* SIMD instructions search 16 bytes at a time, checking: this bucket is not empty, this bucket's key matches the first 7 bits of my key

* Since 16 buckets are checked for emptiness at the same time, you can avoid creating a tombstone for a bucket if any of the other 15 buckets are empty (just set it to empty, i.e. set the first bit to 0)

* This means that tombstones are _very_ unlikely- you'll probably rehash before you get to the load factor where you start seeing tombstones

* Since tombstones are so unlikely, it's more valuable to add an extra bit to the hash prefix than it is to quickly find tombstones

My question remains: why can't the first search return the offset of the first empty bucket? In this loop, why is there not an else that saves the offset?: [https://github.com/abseil/abseil- cpp/blob/256be563447a315f2a...](https://github.com/abseil/abseil- cpp/blob/256be563447a315f2a7993ec669460ba475fa86a/absl/container/internal/raw_hash_set.h#L1652)

<comment><USER19> Ok, I got it. They're exactly the same. Either way you'd need to do a second search, because you're trying to differentiate between 3 states: "probably a match", "empty", or "deleted". A much better way than stealing a bit from the hash prefix is using a special value that represents "empty or deleted", and that's exactly what SwissTable does: [https://github.com/abseil/abseil- cpp/blob/256be563447a315f2a...](https://github.com/abseil/abseil- cpp/blob/256be563447a315f2a7993ec669460ba475fa86a/absl/container/internal/raw_hash_set.h#L253-L278)

</comment></comment><comment><USER20> All good points in the article.

> it was doing something that was so offensive to people who care about > collection performance

Hmm. It also helps here to go back to academia. Big O notation doesn't usually express coefficients/constants, it usually only deals with exponents.[1] The Wikipedia page has a good explanation as to why.

Opinion: coefficients/constants are, however, useful if you're running over a network or some other latency-bound operation.

[1]: [https://en.wikipedia.org/wiki/Big_O_notation#Properties](https://en.wikipedia.org/wiki/Big_O_notation#Properties)

<comment><USER21> The whole point of the article is that constant factors work in Hashbrown's favour.

</comment></comment><comment><USER22> What's the memory efficiency compared to the previous implementation? AIUI robin hood hashing with backwards shift deletion could achieve rather high load ratios, and thus keep memory footprint small. What I read about tombstone based open addressing suggests that it requires a lower load factor and thus more memory.

</comment><comment><USER23> Another alternative: Robind Hood Hashing

[http://norswap.com/robin-hood-hashing-jvm/](http://norswap.com/robin-hood- hashing-jvm/)

(Might actually be the same technique, it's not quite clear!)

<comment><USER24> Uh...this is what Rust uses currently. ;)

</comment></comment><comment><USER25> So how does this degrade for bigger hash tables? Surely the two loops implementation is less efficient since your cache is trashed by the time you do the second look up

<comment><AUTHOR> This varies not with the size of the whole hash table, but with the distance from any given index to the first empty bucket. By keeping the load factor constant, you can grow the hash table as much as you want without degrading the expected performance.

<comment><USER25> Good point, thanks!

</comment></comment></comment><comment><USER26> TL;DR: You first need to look for the hash "tombstone", then you need to find the actual insertion which may be very far away.

Raymond Hettinger (HN commenter and all around Smart Dude (tm)) has a good talk from PyCon 2017 about the evolution of Python dictionaries that goes into tombstones and so much more: [http://www.youtube.com/watch?v=npw4s1QTmPg](http://www.youtube.com/watch?v=npw4s1QTmPg)

</comment><comment><USER27> > . For instance, when compiled with sse2 support it can search 16 buckets at > once.

Does Neon get similar order of magnitude benefit?

</comment><comment><USER28> I wonder if there is any research into hardware architectures optimized for Rust.

<comment><USER29> The things Rust wants more than other languages are support for fast array bounds checks and fast integer overflow checks.

Unfortunately none of the current popular architectures seem to have either.

<comment><USER30> x86 seems reasonable to me:

Fast array bounds checking is, handling the lower bound by unsigned integer math :

      cmp index,bound
      ja crash_handler

Overflow handling is mostly

       jo crash_handler

That's 1 or 2 instructions and the jump has perfect prediction. I'd assume the cost is negligible compared with jump mispredicts and cache misses.

I read somewhere the main problem is LLVM not being quite capable of optimizing these decently: It has to handle all these extra jumps which causes lots of extra basic blocks.

<comment><USER31> I seem to recall, that at least jo has terrible pipelining implications because of the dependency on the flag register that way, no idea about the bounds check (might be a better way)

<comment><USER32> I don't see why JO would be any different from other Jcc, especially since JL is SF ≠ OF. Maybe you were thinking of the problem with INC/DEC and CF, but even then that's a problem of INC/DEC, not of Jcc.

<comment><USER33> It's not usually a big difference, but on modern Intel there is a some difference in performance between the different CMP/JCC options. The more common ones will "fuse" with the CMP and be executed as a single µop, but the rarer ones (like JO and JS) do not fuse with CMP, and thus can add a cycle of delay (and have the overhead associated with executing another µop). The optimization is called "macro op fusion". Details here [https://en.wikichip.org/wiki/macro- operation_fusion](https://en.wikichip.org/wiki/macro-operation_fusion) and here [https://www.agner.org/optimize/microarchitecture.pdf](https://www.agner.org/optimize/microarchitecture.pdf) (pages 108 and 125).

<comment><USER34> Fixing that wouldn't seem to require any _architectural_ changes in x86 though, it's just that Intel hasn't cared enough about JO and friends to optimize them this way.</comment></comment></comment></comment></comment></comment></comment>


You Know WSL Is Bad for GNU/Linux Because Anti-Linux People, Microsoft - <AUTHOR> http://techrights.org/2019/12/08/popaganda/ <comment><USER2> Oh wow. I haven't read such a hard core conspiracy theory article full of wilful words (such as "a Microsoft agent" etc) since decades. When I was young, I shared this view of big bad companies willing to eat up the free will and our beloved GNU/Linux.

Actually, the world is different today. My feeling is that MS is quite struggling with the pure existence of their OS. _Every_ developer I know used Mac OS X of Linux on his machine. So (s)he most certainly doesn't produce a product for the Windows platform. My feeling is that WSL is an attempt to attract these developers. And hell yes, it may work and may reduce the market share of Linux on the desktop. But the "year of Linux desktop" never came and probably never comes, and the OSS community is strong enough to accept Windows as another non-free Linux-distribution. The other (real) distributions won't die from it.</comment>


Warren Buffett on what went wrong with Lehman and Merrill (2003) - <AUTHOR> http://www.forbes.com/2003/05/09/cx_aw_0509derivatives.html

<comment><USER2> Companies and technologies come and go; this year's derivatives are yesteryear's junk bonds.

It's not the technology, but rather a failed system of incentives that leaves mis-managed firms to die from exposure.

<comment><USER3> a failed system of incentives isn't just responsible for this, it's responsible for most of societies avoidable ills. if we would just own up to the reality of many situations and start providing proper incentives we could engineer away many problems.

what incentive does a cop have to stop crime when he makes more money and is safer busting stop sign violators? what incentive does an elected official have to work for the good of his constituents when he will be out of office soon? what incentive do parents have to raise decent children when retirement is socialized and the government baby sits for them? what incentive do businesses have to compete fairly if the government offers easy outs and chances at monopolization (telcos).

<comment><USER4> > what incentive does an elected official have to work for the good of his > constituents when he will be out of office soon?

Surely you're not suggesting longer terms.

Congress critters tend to stay in office a very long time.

<comment><AUTHOR> It doesn't matter how long they stay in office, as long as they don't do counterproductive things to stay in office. Most corporations have a "One share, one vote" policy, so people tend to have a long tenure if they're good at what they do. If corporations switched to the democratic "One man, one vote" policy, you'd expect that, prior to any election, they'd just redistribute some money from their big institutions to lots of individual investors.

<comment><USER5> How would you suggest democratic reform be completed then, votes depending on income? Surely you can see how utterly disastrous that would be?

<comment><AUTHOR> I think democracy is a disaster. Votes based on income would be silly -- what about people who own lots of non income-producing assets, like timber, minerals, early-stage companies, etc? Surely a system of votes based on wealth would be smarter.

The governing business should operate like the cereal business, or the software business, or the anything-but-government business: people pay a reasonable price for a particular service, and get what they pay for. You'll notice that a company like Google is somehow able to make its users happy, even though it doesn't have monopoly power on multiple industries or the ability to print arbitrary amounts of dollars, and yet the government is _unprofitable_ even with these advantages.

Just tell me how you'd convince Larry and Sergei (and their shareholders) to turn Google into a democracy, and I'll concede that perhaps democracy is a sensible system after all.

<comment><USER6> Except that government, almost by definition, deals with goods that necessarily involve lots of externalities, involve a public good, or require transaction costs too great for any one entity to pay them. One person's service is another person's annoyance.

Think about the highway system. Say that people paid a certain amount for the privilege of driving on the roads. But then what about the folks whose land was taken for the highway? How about the ones who now have to put up with the noise of thousands of automobiles? And then what about positive externalities - the business owners who make more money because cars can get to their businesses, but don't pay to use the roads themselves?

Or think about the legal system, including consumer protection agencies. What happens if someone contracts for a service, and then never pays up? Maybe enforcement of service contracts should itself be a service. But then what if that service itself breaches its agreement? Who do you go to?

Democracy's a pretty shitty solution, but it exists for a reason. All the alternatives seem to be worse.

<comment><AUTHOR> _Except that government, almost by definition, deals with goods that necessarily involve lots of externalities, involve a public good, or require transaction costs too great for any one entity to pay them._

Interesting. Most of the big expenses my government pays seem to involve some sort annuity scheme in which everyone is required to participate in the same retirement plan, even though the plan doesn't have nearly enough capital to handle its obligations. They also do health care, and they blow stuff up. It seems to me that most of the externality-based arguments can be taken care of with insurance and derivatives markets.

For example, if we decide it would be a Good Policy to make everyone eat an apple a day, because it would increase life expectancies by one year on average, we could just mandate that they all do it. But we could also make a bet on the average life expectancy in the US, pay people to eat apples, and pocket the cost differential. I'm not sure the transaction costs would be lower, but 1) the incentive to lower costs would be there -- no bureaucrat DMV ever got a seven-figure bonus for cutting costs, but plenty of bankers, consultants, and private equity executives have, and 2) it would have lower exit costs than our current democratic experiment.

 _But then what if that service itself breaches its agreement? Who do you go to?_

David Friedman claims that they'd form little federations predicated on adhering to certain standards. If one company didn't follow through, it would have to renegotiate separate treaties with every other enforcement agency (and would lose lots of goodwill).

 _Democracy's a pretty shitty solution, but it exists for a reason. All the alternatives seem to be worse._

Yes, yes, and no. It would be pretty weird for Democracy to be exactly the wrong way to run almost every human institution, even though almost every human institution is run better than the average democracy.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER7> "Derivatives are financial weapons of mass destruction, carrying dangers that, while now latent, are potentially lethal."

Prescient.

</comment><comment><USER8> It's informative how badly wrong Greenspan was, compared to Buffett.

<comment><USER9> One is dealing with his own money while the other is not.

<comment><USER10> Another important point: One is much closer to anarcho-capitalism in philosophy than the other.

 _edit: if you downmod me, please explain why. this is a great opportunity to learn._

<comment><USER11> I suggest you explain your own comment in the first place. Who is closer to anarcho-capitalism in philosophy? The former central banker or the guy who wants to increase taxes?

</comment></comment></comment></comment><comment><USER12> The intuition of the man is baffling.</comment>


Show HN: win-notify: simple notifications API for JavaScript Windows Store apps - <AUTHOR> http://tadeuszwojcik.github.io/win-notify/

<comment><AUTHOR> Any feedback welcome :)</comment>


Ask HN:How can a non-technical guy find/screen mobile/tech talent - <AUTHOR>

I know this is a variant on a theme but hoping you can suggest ways to screen technical talent. I am confident we can identify the fire in the belly, the passion and the raw curiosity that are important measures of success. I am less sure of our ability to discern true hacker talent for our start up. We are a modestly funded group of folks launching a db driven mobile app. We are ideally looking for someone locally, in the Puget Sound area and have found some great resumes but have not been able to find the right mix of passion and comfort with the other founders that this person can deliver the goods. Any suggestions (private responses welcome via the email in my profile as well). <comment><USER2> I recommend using a lot of references, and looking for specific examples of projects the person has worked on. Find out exactly what they did for a project, and make sure they can speak about it well. If they aren't able to point to specifics look for another candidate.</comment>


Debugging across pipes and sockets with strace [pdf] - <AUTHOR> https://github.com/nh2/strace-pipes-presentation/blob/master/presentation/Debugging%20across%20pipes%20and%20sockets%20with%20strace.pdf <comment><USER2> "95% of all computer problems can be solved with strace" \-- if you do system/network programming, probably yes, if you want to master Microsoft Word, probably not.

Side question: How did the author create the PDF from the markdown source at [https://github.com/nh2/strace-pipes- presentation/blob/master...](https://github.com/nh2/strace-pipes- presentation/blob/master/presentation/Debugging%20across%20pipes%20and%20sockets%20with%20strace.md) ?

<comment><USER3> > _How did the author create the PDF from the markdown source_

He used a tool called Marp [1].

At the top of the Markdown file you can see a code that reads "page_number: true" which can be found in the sample file provided by Marp here [2]. There are other tools out there that can be used to convert a Markdown into slides or PDF. I personally use Deckset.app [3] but searching _" markdown presentation"_ — or — _" markdown slides"_ on the Internet seems to bring a long list of projects aiming for the same feature set.

[1] [https://yhatt.github.io/marp/](https://yhatt.github.io/marp/)

[2] [https://raw.githubusercontent.com/yhatt/marp/master/example....](https://raw.githubusercontent.com/yhatt/marp/master/example.md)

[3] [https://www.decksetapp.com/2/](https://www.decksetapp.com/2/)

<comment><USER4> > There are other tools out there that can be used to convert a Markdown into > slides or PDF

For example Pandoc which I personally enjoy

</comment></comment></comment><comment><USER5> I made a little tool some time ago to show the relations of processes, fifos, pipes and sockets in a handy picture. This turns out to be very handy when debugging these kind of problems:

[https://github.com/zevv/lsofgraph](https://github.com/zevv/lsofgraph)

<comment><USER6> This looks cool! I tried to use it, but I don't have "dot". It's kind of hard to search for, can you point me in the right direction?

<comment><USER7> dot - "hierarchical" or layered drawings of directed graphs. This is the default tool to use if edges have directionality.

[http://www.graphviz.org](http://www.graphviz.org)

</comment></comment></comment><comment><USER8> Markdown versin: [https://github.com/nh2/strace-pipes- presentation/blob/master...](https://github.com/nh2/strace-pipes- presentation/blob/master/presentation/Debugging%20across%20pipes%20and%20sockets%20with%20strace.md)</comment>


MIT Researchers Develop Computer-Generated TCP That is Twice as Fast - <AUTHOR> http://web.mit.edu/remy

<comment><USER2> Looks very interesting.

So we if we imagine this gains traction, could a mobile device use a different transport protocol for different network situations? Such as a phone going from 3G to 4G to wifi?

I guess the problem is the other end would have to adapt too?

Also if you are going to add new protocols, then would be not be better to add more message oriented protocols, rather than stream oriented? As many applications (twitter, payment instruction, email send) are message oriented.

</comment><comment><USER3> Bit of a misleading headline:

>> For example, with eight senders sharing a 15 Mbps link, a >> Remy-generated algorithm achieved more than twice the >> median throughput, with less than half the queueing delay, >> of Compound TCP and TCP NewReno. It achieved 70% more >> throughput than TCP Cubic and more than a threefold reduction >> in queueing delay.

<comment><USER4> People have tried to start this discussion already:

[https://news.ycombinator.com/item?id=6087828](https://news.ycombinator.com/item?id=6087828)

[https://news.ycombinator.com/item?id=6079079](https://news.ycombinator.com/item?id=6079079)

[https://news.ycombinator.com/item?id=6072298](https://news.ycombinator.com/item?id=6072298)

[https://news.ycombinator.com/item?id=6072086](https://news.ycombinator.com/item?id=6072086)

</comment></comment><comment><USER5> Cool.</comment>


White House Code Sprint contest seeks job search apps for young people - <AUTHOR> http://www.informationweek.com/thebrainyard/news/232800224/white-house-seeks-summer-jobs-apps

<comment><USER2> Looks like the deadline was 3 days ago.</comment>


Why Is Metadata Management Important? - <AUTHOR> https://metatron.app/2019/03/29/why-is-metadata-management-important/ <comment><USER2> It's an interesting subject. What are the best MDM tools?

<comment><USER3> Take a look at [https://github.com/metatron-app/metatron- discovery](https://github.com/metatron-app/metatron-discovery). They have a simple and quick version of MDM feature.</comment></comment>


Do C Callbacks Like This. Not Like That - <AUTHOR> http://www.mindtribe.com/2015/06/do-c-callbacks-like-this-not-like-that/

<comment><USER2> great piece: informative and well-written!</comment>


Inside Google's Secret Drone-Delivery Program - <AUTHOR> http://www.theatlantic.com/technology/archive/2014/08/inside-googles-secret-drone-delivery-program/379306/?single_page=true

<comment><USER2> This looks much better than a "traditional" helicopter style drone that Amazon is demonstrating. Interesting that Google picks this up as well, probably we are going to see lots of drones doing deliveries in 5 years.

</comment><comment><USER3> Of all the press about this project today, this is by far and away the best article.

<comment><AUTHOR> It's rather interesting that it hasn't made the frontpage.</comment></comment>


Freshmen make breakthrough discovery on 30 year old astronomy problem - <AUTHOR> https://uniavisen.dk/en/solved-astronomy-mystery-after-just-one-year-at-university/ <comment><USER2> I'm confused. What's the 30-year-old problem?

Whether there is a classification of gamma ray bursts or two classes or predicting in what class a burst belongs into?</comment>


HNLive – Hacker News in Real Time - <AUTHOR> http://hnlive.cf/newest <comment><USER2> Doesn't work, only shows (in emacs w3m):

                                                     HN Live
                                      live updating version of hacker hews
                                             hot new show ask about
                                                Fetching data...

If you want a good real time feed, use irc://irc.freenode.org/#hn</comment>


The Future of Apache Beam, Now a Top-Level Apache Software Foundation Project - <AUTHOR> https://www.talend.com/blog/2017/01/13/future-apache-beam-now-top-level-apache-software-foundation-project/ <comment><USER2> Related: I've been collecting this and all other relevant articles about Apache Beam on [https://reddit.com/r/dataflow/](https://reddit.com/r/dataflow/).

Disclaimer: I'm Felipe Hoffa, and I work for Google Cloud. [http://twitter.com/felipehoffa](http://twitter.com/felipehoffa)

(not using yet [https://reddit.com/r/apachebeam/](https://reddit.com/r/apachebeam/) as the combined volume is still too low)

</comment><comment><USER3> Digging into beam, is there a example where one loads csv data, does some basic transformations and sends results to bigquery?</comment>