Google's end-to-end key distribution proposal - <AUTHOR> https://code.google.com/p/end-to-end/wiki/KeyDistribution <comment><USER2> The first comment by Mailpile seems to highlight the biggest problem to me:

>Hello! Bjarni from the Mailpile team here.

>This is an interesting proposal and sounds like a significant improvement over the current centralized key-server model.

>The main quibble I have with it, is it seems there's no concern given to the privacy of user's communications - the proposed gossip mechanism seems designed to indiscriminately tell everyone who I am communicating with. That's a pretty severe privacy breach if you ask me, worse even then PGP's web-of- trust because it's real-time, current metadata about who is interacting with whom.

>Am I misunderstanding anything here?

>\- Bjarni

<comment><USER3> As I just posted as a comment to the proposal:

In the EU, e-mail addresses are personally identifiable information. It's not clear that an append-only log with no expiration or means for individuals to delete the content will even be legal in many EU countries.

<comment><USER4> What do you mean? When users sign up to a Key Registry, they are explicitly acknowledging that their emails and public keys can be used in the way described by the protocol. AFAIK, that's perfectly legal.

<comment><USER5> in EU, you can say at any point to a company 'delete every piece of personal information about me that you have', and email is considered an personal information. Even if you gave consent, you can withdraw consent at any point in time.

<comment><USER4> So, basically, I can tell my bank/credit agency/hospital "please forget that I got a mortgage/defaulted/tested positive for HIV"? Doesn't that sound rather risky?

<comment><USER6> It's not that simple (in the UK at least), you only have to stop processing information if "unwarranted and substantial damage or distress" is being caused.

<comment><USER7> Additionally, all stop processing / deletion regulations require a court order, which will be subject to all the usual requirements of justice.

<comment><USER3> It only takes a single successful demand before an append-only log because untenable.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER8> I was amused to see that on neither of the links in the post does SSL work correctly. convergence.io identifies as whispersystems and www.certificate- transparency.org produces an ssl error on every version of openssl and browser I have to hand (I didn't try anything with non-standard ECC curves).

That said, however, the entire end-to-end project is for me one of the most interesting and exciting practical innovations in security in years.

</comment><comment><USER9> Interesting. It has bugged me for some time that if the Web of Trust was bigger, it could grow exponentially and become universal: once someone you personally know has entered you into the WoT, you can be trusted and can trust others based on a number of public signatures on their public key. However, currently the WoT is so sparse that you cannot do this.

My idea was to use the existing web TLS platform to bootstrap the WoT to a sufficiently large level. I run my email on my own domain. Why can't I tell the WoT (and have it trust that it is true) that my public key is XYZ by putting it at [https://igorpartola.com/pub.pem](https://igorpartola.com/pub.pem)? GMail could do something similar and at least to start, we could get enough emails validated to start having the WoT spread on its own. Then we could modify the infrastructure to remove the public CA's and central authority entirely, by using the WoT itself. Google's HTTPS cert would then be based on its PGP key and be verified by humans inside the WoT.

I also think that the important part of the WoT is verifying emails/digital identity, not government docs. I don't care if I am talking to "Bob Bobber", I care that I am talking to firstname.lastname@example.com. I may never have met email@example.com, but I see his/her public git repos, blog, etc. and I want to connect to them securely.

<comment><USER10> Let's not forget that one of the most successful attack vectors is social engineering, that is, tricking people into trusting you and making you part of the WoT.

<comment><USER9> Sure. But if you are willing to trust some CA to issue me a HTTPS cert for my domain, why are you not willing to trust that I serve my public key for this domain suing this HTTPS cert to secure its transport? Oh, sure some adversary could gain control of the web server used for this and replace my pub.pem, but then I will notice and revoke it. And once enough people download and sign my pub.pem, it no longer matters: I am now in the WoT and can remove pub.pem.

<comment><USER11> I think the issue is that a lot of people who are informed about CAs _don 't_ trust them, but we still don't have anything better that's anywhere even close to wide adoption.

</comment></comment></comment></comment><comment><USER12> Reading through the spec, there is something eerily familiar with the key directory implementation. Quoting:

 _Alice will obtain a proof from the Key Directory that demonstrates that the data is in the permanent append-only log, and then just encrypt to it.

Within the message to send to Bob, Alice includes a super-compressed version of the Key Directories that fits in a 140 characters (called STHs which stands for Signed Tree Heads). This super-compressed version can be used later on by anyone to confirm that the version of the Key Directory that Alice saw is the same as they do_

Append-only log. Global. Hash of the log's tip state at the time of use...

Smells like a mixed blockchain/git type approach - which is a good thing. The "super-compressed" version of the log tip sounds like git revision hash. The append-only, globally distributed log is pretty much like a blockchain.

And it attempts to solve a really hard global problem. I like it.

<comment><USER13> The blockchain (as most people understand it atleast) is an implementation of a Merkle tree. Which is why Git/Bitcoin are eerily similar - they both utilize Merkle trees for integrity.

The real innovation in the "blockchain" was using proof of work in combination with the Merkle tree in order to enforce a single history. Take that away and yes, it looks alot like Git. :)

<comment><USER14> And if you consider that a git commit is actually "proof of work" and find a way to quantify the value of that commit (fixes issue x which was worth y points, passes all regression tests), you would have... gitcoin.

<comment><USER11> Quickly, buy the .in domain name and throw a client up onto github ;)

Edit: Domain already taken; I guess after bitcoin's success people just bought up (word)co.in for every value of word they could think of...

<comment><USER14> I'm only half joking :) Gitcoin is much looser than bitcoin when it comes to verifying proof of work. However, there is a (practical) sense in which it exists. Fixes are submitted, accepted, merged, pulled until they are part of all users' "blockchains". gitcoin is the quantization of this karma. The lack of a hard definition of work means that it is perhaps easier to bootstrap in a centralized ecosystem like Github. Like Quora credits for code.

In much the same way as Quora credits are used to power A2As, gitcoin would enable you to ask J Random Coder on github for a fix and pay him in karma. The GPL is effectively a no-freeloaders mechanism, gitcoin could be another.

</comment></comment></comment></comment></comment><comment><USER15>
      The special thing about this Key Directory, is that
      whatever is written in the directory can never be
      modified, that is, it's impossible to modify anything from
      there without having to bring the service down, or telling
      to everyone that's looking about what is being modified.

The MIT PGP key server has eleven different keys in my name which I created in 1997, when I was about 12 years old. Of course, I have long since lost the private keys and e-mail addresses.

I guess with this proposal, the fact you used to go by firstname.lastname@example.com will be part of your permanent record.

<comment><USER11> Unless I missed a part of it, the key is only permanently tied to the email address - i.e. you could always say that email@example.com is a different Joe Bloggs and not you (probably only plausible if you have a common to medium rarity name though).

</comment></comment><comment><USER16> My suggestion is for them to work at least another 12 months on this, before they even begin to publicly test it on Gmail accounts. We need more privacy quickly, but let's not rush into a protocol that could last for another 20 years, if all the major e-mail providers adopt it. We need to get this right.

I wish the DIME (former Dark Mail) protocol was out already, too, so we can compare. There's also Adam Langley's Pond [1], but sounds like it's too complex, and it only works over Tor. And TextSecure/Axolotl could probably be used as a secure e-mail protocol, too, if you add a proper e-mail like interface. I hope the team behind End-to-End is looking at all of them.

[1] - [https://pond.imperialviolet.org/](https://pond.imperialviolet.org/)

</comment><comment><USER17> I have been through this thought process before. The conclusion I came too was that the implementations should be transparent, but that the user information should not.

Basically I was not going to put up a list of everyones email addresses and keys anywhere, and certainly not who they connect with.

The more I looked into the problem, the more I realised that the vast majority of users would rather sacrifice security for usability. Even in my implementation people would rather not see the "Please verify this key with the recipient" page. They just want to get something done. I think this proposal from google would work well so long as their base implementation involves no additional steps beyond that of a normal email client.

My implementation uses a central key authority, however the application is pure javascript, and the entire javascript is downloaded to the browser prior to the user entering their email address and password. After that no more code gets sent to the client. You can verify it wont steal your data.

I have the same problem of initial key exchange that everyone else does, but I give the user options to verify they keys themselves. Once they have they encrypt their own contact list (along with keys) and re-upload it. Therefore limiting the attack vector to initial key exchange.

If anyone wants to have a look check out [http://senditonthenet.com/](http://senditonthenet.com/)

<comment><USER18> Do you use Key Escrow for private key storage? How can the other receiver decrypt the file using his browser only? Where do you store the private key?

<comment><USER17> The users password acts as a symmetric key. It is never sent to the server, but a hashed copy is sent to the server for authentication, which is then rehashed and stored in the DB.

The users private key is AES encrypted with the password as key and sent to the server for storage. A JSON hash of their contacts is also encrypted in the same way and sent to the server for storage.

</comment></comment></comment><comment><USER19> I hope this works out. If enough of the big email vendors (gmail, outlook, yahoo, etc.) get on board the network effect could be enough to push adoption to a very high percentage.

Once that happens my "unencrypted" email folder would be viewed about as often as my junk mail folder.

And then... maybe just maybe... spammers will be faced with a serious challenge.

<comment><USER20> > And then... maybe just maybe... spammers will be faced with a serious > challenge.

Why? Couldn't spammers encrypt the mail they send to you just like everybody else?

<comment><USER21> Because, I think, they would have to be part of the Key Directory too, listed under "Spammer".

This kind if implies a secondary directory market of ratings and rankings of the prime source (ie isASpammer, isWithinThreeHopsOnLinkedIn)

<comment><USER3> They'd just keep creating new keys, and use a botnet to exchange seemingly valid e-mails with itself and a trickle of seemingly legit e-mails (buy access to a few small legit mailing lists to make sure their botnet isn't a total island). It's not clear to me that this won't make it possible for spammers to manufacture/buy sufficient "trust" to put themselves in a better situation than they're in today.

<comment><USER21> Darn - I think you are right... Ah well, defeating spam is only a nice to have from this

</comment></comment></comment></comment></comment><comment><USER22> How will the Key Directories and third party Monitors verify that I'm the real owner of my self-hosted email address firstname.lastname@example.com, uploading my real public key to the directory?

<comment><USER23> Depends on what you mean by "verify the real owner". They can verify that whoever has control of what gets uploaded to myowndomain.com also has control of the firstname.lastname@example.com email account and can prove they have the private key corresponding to the public key that was posted. Is that what you meant?

<comment><USER22> Is that verification process described somewhere in Google's proposal? I don't understand how it would work with third party Monitors. Would I have to prove my identity with some email-callback or DNS/HTTP token separately to them all?

</comment></comment></comment><comment><USER24> Wouldn't proper compartmentalization dictate that email providers be explicitly eliminated from the end-to-end encryption process?

Apart from being able to inspect email and recognize that it contains content that looks like an encrypted form of something, I think we wouldn't want them to be explicitly informed that encryption was used, or know anything about the encryption algorithm, or know anything about how keys were distributed, or see any keys even public ones.

I think this would apply to the general case where someone uses an email service that is run by another party. In the common cases of major email providers with business models that conflict with privacy and security in various ways, the risks would higher. Even before factoring in their being high priority targets for hacking, government surveillance of questionable legality, etc.

</comment><comment><USER25> TLDR is: _Distributed (as in DNS) directory with append-only entries keyed by hash-of-email and third party replication /validation/logging._

Discussion of spam implications at [https://moderncrypto.org/mail- archive/messaging/2014/000727....](https://moderncrypto.org/mail- archive/messaging/2014/000727.html)

Widespread encryption _could_ be a recipe for Google (as the largest of the proposed directories) sliding in an identity reputation scheme (eg. based on location / communications history) and brokering it as a service.

</comment><comment><USER11> So it seems they invented PGP keyservers with a monitoring protocol as a bag on the side?

I've had keys on keyservers for years. The monitoring side is interesting though.

It's also unclear how the whole directory will be compressed to 140 bytes - iirc, the best compression algorithms reduce text by ~80-90%, so it might work for a week or so, I guess.

<comment><USER26> Presumably the 140 characters is more like a git revision number than reversible compression.

</comment></comment><comment><USER27> _" The model of a key server with a transparency backend is based on the premise that a user is willing to trust the security of a centralized service, as long as it is subject to public scrutiny, and that can be easily discovered if it's compromised (so it is still possible to compromise the user's account, but the user will be able to know that as soon as possible)."_

So it's a central service where we can - eventually - find out if a key changes or is invalid, though not necessarily if anyone just breaks into the service. An attacker can still break in and monitor authentications to gather intel on users. Or they can automate an attack such that the keys are compromised and the attacker gets the access they want while you're asleep at 3am.

 _" a Monitor could notify the user whenever a key is rotated, and the user should be given a chance to revoke the key, and rotate it once again."_

Now the user needs to see this e-mail about their key getting rotated, realize it's a compromise, issue a new rotation, and hope nothing bad happened in the meantime.

 _" making it so hard to hide a compromised key directory, it would almost require shutting down all internet access to the targeted victim."_

This is completely within the scope of many different types of attacks used today, though often you only have to limit it to specific servers.

 _" The model envisioned in this document still relies on users being able to keep their account secure (against phishing for example) and their devices secure (against malware for example), and simply provides an easy-to-use key discovery mechanism on top of the user's existing account. For users with special security needs, we simply recommend they verify fingerprints manually, and we might make that easier in the future (with video or chat for example)."_

So it really is just a PGP keyserver. For users who care about security, do something else to make yourselves more secure.

</comment><comment><USER28> Slightly worrying to see the word "checksum" used to refer to a one-way cryptographic hash function...

<comment><USER29> Huh? The only mention of a checksum also acknowledges it's possible to brute- force reverse it, aka not one-way.

<comment><USER28> A checksum is a computationally efficient way to detect _unintentional_ changes in data. But as protection against _intentional_ changes it's quite useless. That's what cryptographic hash functions are for.

Now the proposal described using a "checksum" to tie together two halves of a self-signature, where one half would be a an identity and the second would be an email address, essentially. That would make it trivial to forge a second half with another email address (but the same "checksum").

I understand that's an entirely different problem. All I'm saying is that few cryptographers would use the word "checksum" at all, and even fewer would use it in this context. That's what worries me slightly.

</comment></comment></comment><comment><USER30> The last people in the world you want managing your keys. Why don't you just send them directly to the NSA ?

<comment><USER11> As much as I generally distrust google with privacy when it comes to actually handing over data, in this case it's an open protocol that has just been drafted by google; anyone can run a server or make their own implementation. It also never touches private keys, only public. If you want a public key to have limited distribution, you don't put it on a keyserver, and instead only exchange it with the people you would like to in person or over some other verified and secure channel.

It does have a somewhat anti-privacy feature in that if I understand it correctly, it keeps a record of messages between participants (in the sense of a record existing _that a message was exchanged_ although not the content), but that level of data is already accessible to NSA/GCHQ anyway.

<comment><USER30> You think google can write a proposal like this without the NSA getting involved ? The fact that they are preserving metadata is revealing and meaningful. Its another PR stunt.

Google can never never be trusted again. They publicly lied about PRISM, and they got caught. These people have no business making security protocols for us.

<comment><USER31> What statement did Google make, regarding PRISM, that was a lie? And what evidence caused them to get caught?

<comment><USER30> [http://www.cbsnews.com/news/google-ceo-larry-page-issues- sta...](http://www.cbsnews.com/news/google-ceo-larry-page-issues-statement-on- prism/)

[http://www.washingtonpost.com/wp- srv/special/politics/prism-...](http://www.washingtonpost.com/wp- srv/special/politics/prism-collection-documents/m/)

[http://gizmodo.com/confirmed-nsa-paid-google-microsoft- other...](http://gizmodo.com/confirmed-nsa-paid-google-microsoft-others- millions-1188615332)

They are lying through their teeth. As you should know skj, being a google employee.

<comment><USER31> What is the actual lie?

<comment><USER32> More to the point, what is the truth? Google leadership post 9/11 knew what "certifying" their communication systems meant and took the money to do it. Whether they knew that the program was named Prism is irrelevant, in my opinion.

<comment><USER31> Google responds to legal requests for information. The whole PRISM scandal was indicating that the NSA had some kind of direct link into the databases themselves. This assertion is what Google denied, and still denies.

There's been a bit of a wandering definition for PRISM, from "they have NSA software with root access to all machines!" to "they receive LEO requests for information, which they review, and sometimes fulfill." The former is untrue, and the latter is true.

<comment><USER33> > The whole PRISM scandal was indicating that the NSA had some kind of direct > link into the databases themselves.

Though, as Google and Yahoo later found out, the NSA was directly tapping cables between their data centers [1].

[1] [http://www.wired.com/2013/10/nsa-hacked-yahoo-google- cables/](http://www.wired.com/2013/10/nsa-hacked-yahoo-google-cables/)

<comment><USER31> That appears to be several things:

1) True.

2) Consistent with statements made by Google.

3) Quickly mitigated by Google when it began encrypting traffic between data centers.</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment>


Photo Essay: Unlikely Places Where 'Wired' Pioneers Had Their Eureka! Moments - <AUTHOR> http://www.wired.com/culture/lifestyle/multimedia/2008/03/ff_eureka?slide=2&slideView=2 <comment><USER2> Interesting, but not really all that unlikely. At home? At work? On a train? At lunch?

</comment><comment><USER3> Cornell has like 15 cafeterias. Anybody know which one was the plate-wobbling one?</comment>


Show HN: A hierarchical Markdown notetaking tool with mouse/tablet drawing - <AUTHOR> https://github.com/blackhole89/notekit/ <comment><USER2> Just reading the description makes me wonder if I should switch from Mac to a pc laptop with a digitizer in the screen.

Great concept, best of luck to the developer!

</comment><comment><USER3> This ticks a great set of checkboxes but at least for me it misses the critical ones because it can't be run on iOS or Android devices.

<comment><AUTHOR> As I see it, the expectations I'd have towards a mobile note-taking tool (like Google Keep) are different enough that I think it should be a separate program. Folder structure would be fairly cumbersome to navigate on a cellphone, and few keyboards make it particularly easy to type the symbols that control Markdown styling; on the other hand, you'd probably really want solid support for shopping/TODO lists and the like, for which Google's tiling organisation paradigm seems very useful.</comment></comment>


Sorry about the fire in your home, I learn more. - <AUTHOR> http://www.translateparty.com

<comment><AUTHOR> (From the starting sentence "You'll pardon me for mentioning it, but your house is on fire.")</comment>


Finding the perfect office chair - <AUTHOR> http://www.fourhourworkweek.com/blog/2009/01/27/office-chair-aeron-vs-mirra-vs-liberty/

<comment><USER2> Aeron's are my chair of choice. I started using one at my job and it's amazing how much more comfortable it is than the one I have at home. I've begun actively hunting for one to get for the house, despite the prohibitive cost they're just amazing.

</comment><comment><USER3> Aeron chair is the best. Too bad its so expensive and can't carry it with me so it stays in Texas even when I'm across the country :(</comment>


How one woman's name caused massive issues at a large telecom operator - <AUTHOR> http://irhadbabic.com/its-all-her-parents-fault/ <comment><USER2> Cache:

[http://webcache.googleusercontent.com/search?q=cache:mCCbjRY...](http://webcache.googleusercontent.com/search?q=cache:mCCbjRY85bwJ:irhadbabic.com/its- all-her-parents-fault/+&cd=2&hl=en&ct=clnk)

Obligatory XKCD: [https://xkcd.com/327/](https://xkcd.com/327/)

And no, it's not her parents' fault, it's the fault of the unbelievably (actually, sadly, all too believably) bad programmers.

Good programmers learn from their mistakes, the best programmers learn from other people's mistakes. Read widely, very widely, and learn.</comment>


I Don't Give a Damn About Getting Into YC - <AUTHOR> http://blog.mahesh.io/2012/10/29/i-dont-give-a-damn-about-getting-into-yc/

<comment><USER2> Why does it bother you that others care about getting into YC? People have different understanding of our world and their positions in that. It is fairly legitimate for them to express what's on their mind and their feeling about their actual priorities.

Having said that, I think you shouldn't care much about getting into YC. One size doesn't fit all.

<comment><AUTHOR> This was not my point: the headline and the whole post were meant to be thought provoking. If I spent time to apply to YC it means that I obviously care about it. And it's good that a program like this attracts a lot of smart people. I just think it is really wrong to pass the message that YC is major goal itself, at the point you need to "hack in" or attend online specific preparation classes. It looks to me this approach just risks to shift too much the baricenter from making things to showing off some standard sexy stuff.</comment></comment>


Web Components: How to Craft Your Own Custom Components - <AUTHOR> https://auth0.com/blog/web-components-how-to-craft-your-own-custom-components/?utm_source=hacker_news&utm_medium=sc&utm_campaign=web_components <comment><USER2> Why bower? Didn't bower fall by the wayside when npm v3 came out a couple years ago?[1]

[1]: [https://www.quora.com/Why-use-Bower-when-there-is- npm](https://www.quora.com/Why-use-Bower-when-there-is-npm)</comment>


Syscall Auditing at Scale - <AUTHOR> https://slack.engineering/syscall-auditing-at-scale-e6a3ca8ac1b8#.1lnibz30j <comment><USER2> As the kind-of-sort-of maintainer of Linux's syscall infrastructure on x86, I have a public service announcement: the syscall auditing infrastructure is awful.

It is inherently buggy in numerous ways. It hardcodes the number of arguments a syscall has incorrectly. It screws up compat handling. It doesn't robustly match entries to returns. It has an utterly broken approach to handling x32 syscalls. It has terrifying code that does bizarre things involving path names (!). It doesn't handle containerization sensibly at all. I wouldn't be at all surprised if it contains major root holes. And last, but certainly not least, it's eminently clear that no one stress tests it.

If you really want to use it for production, invest the effort to fix it, please. (And cc me.) Otherwise do yourself a favor and stay away from it. Use the syscall tracing infrastructure instead.

<comment><USER3> Hi - Ryan from the blog post here.

eBPF is great, and we plan to support it as our collection mechanism, but not today and maybe not soon. When we wrote go-audit, there were few, (if any?), distros that had eBPF kernel support. Now Ubuntu 16.04 has support, as do many others, and that is great!

But..

Releasing a tool that will work for most people today is very different to releasing one that will work for most people in a couple of years. We are sharing this imperfect tool that uses an using an imperfect syscall monitoring mechanism, both of which were written by imperfect humans.

We have found plenty of problems with auditd during development (some real facepalm stuff, to be sure), but it also does many useful things. We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot.

If you don't trust auditd, that's a totally valid opinion to hold. I welcome criticism of our approach, but a PSA that doesn't involve having testing our tool is rather unfair.

<comment><USER4> > When we wrote go-audit, there were few, (if any?), distros that had eBPF > kernel

> support. Now Ubuntu 16.04 has support, as do many others, and that is great!

This! I'm rather tired of people preaching about eBPF when in reality many of us have to run long term supported kernels/ distros like RedHat or Ubuntu LTS releases.

<comment><USER5> Where did amluto "preach" eBPF?

What is said is "If Slack really want to use syscall auditing for production, invest the effort to fix it, please. (And cc me.)", which seems totally reasonable to me.

Slack's answer is "We have worked around limitations and very definitely stress tested it in our environment. A lot. A lot lot." As I understand, this means "No, Slack will not invest any effort to fix Linux syscall auditing in upstream kernel, because we have already worked around limitations." Which is kind of sad and expected.

</comment></comment></comment></comment><comment><USER6> Great idea. I always thought that it's essential to log events in realtime to a remote system that is secure and harder to compromise to modify the logs post-intrusion. Way back in the day it was suggested to do this to an entirely offline system by cutting the rx pins on a parallel cable, thereby only allowing the one-way transmission of logs to the log server. I don't know if anyone ever did that in practice though.

Anyways this invites the question, are you allowing your production servers to make outbound internet connections? Generally, I would proxy outbound connections and/or use internal mirrors and repos for the installation of software.

<comment><USER7> > it was suggested to do this to an entirely offline system by cutting the rx > pins on a parallel cable, thereby only allowing the one-way transmission of > logs to the log server.

sounds like overkill, but pretty cool i must say.

</comment></comment><comment><USER8> Looks like a nice tool, and it's great to see syscalls getting more attention.

I don't fully get the argument for why on-host filtering is undesirable. Of course naively filtering for curl-originated connections isn't a solid detection scheme for rootkit-installs! That's just a naive filter, which a naive user could mis-use in a centralized way or in a distributed way.

As for event correlation (#2 of the pros), it can be done on-host too. And back-testing (#3) of new rules is indeed a highly valuable feature! But you certainly don't have to log everything centrally to get that capability. E.g. in the case of Falco, you can capture trace files and re-run any number of rules/filters on them.

I do agree with the point on rules being exposed to an attacker.

[Disclaimer: author of the initial version of Sysdig Falco]

<comment><USER9> Regarding on-host filtering (edge analytics), my experience has been it's because of performance, and I agree with the security angle, too.

</comment></comment><comment><USER10> The issue with enabling syscall auditing is the overhead it introduces, iirc some around two orders of magnitude, as in 200000/s -> 3000/s. I would just use seccomp-bpf filters on a per program basis as the overhead there according to benchmarks is much less.

<comment><USER11> seccomp-bpf is awsome, but it has to be really baked into the app to be useful. Not being able to filter by deref memory is pretty limiting.

But not all syscalls need to be logged. You don't need to audit all read()/write() calls for example. open/connect/exec _/ set_id should give you plenty of information already. I know at least Fedora had audit included, but inactive, and I haven't heard of any terrible performance degradation there.

(it looks like the penalty was ~40ns in 2014 [http://permalink.gmane.org/gmane.linux.kernel/1639528](http://permalink.gmane.org/gmane.linux.kernel/1639528))

<comment><USER2> It was an order of magnitude higher at one point. These days it's not _that_ bad under most workloads.

</comment></comment></comment><comment><USER9> This is seriously cool, in my opinion. In fact, I've been working on something kind of similar in Go with more support for network monitoring and vulnerability management that would feed into GrayLog.

Are you guys hiring?

</comment><comment><USER12> If you're wondering why this is useful, or if you need it at all, ask yourself if you would want to get an alert when nginx execs a shell, or opens /etc/shadow. Syscall auditing gives you the lowest possible interface to capture these events.

It's not for the faint of the heart - the volume of events to filter through requires some serious infrastructure - but it's an important component of a mature secops program.

<comment><USER2> The even more mature secops programs might wonder whether the kernel syscall auditing infrastructure itself is a problematic attack surface.

</comment></comment><comment><USER13> If you want similar functionality to the first questions without audit, try netfilter. It's still shitty logs, unfortunately, but so is most monitoring.

</comment><comment><USER14> Similar to [https://github.com/gdestuynder/audisp- json](https://github.com/gdestuynder/audisp-json) and [https://github.com/mozilla/audit-go](https://github.com/mozilla/audit-go)

</comment><comment><USER15> Couldn't you just do this in user-land with `PTRACE_SYSEMU`? Then your tracing processing also has to make 1 system call to unlock and allow the other process to run?

I started tooling up a basic version of this but I need to change PTRACE flags, and change from reading /proc/[PID]/mem to `process_vm_readv(2)`

<comment><USER16> You know that ptrace messes up the child parent relationship (this is why you can't strace a strace'd or gdb'd process already. It goes 1 level deep only) and has a serious performance impact, whereas syscall tracing doesn't, right?

</comment></comment><comment><USER17> Would this be useful in a containerized architecture where everything is run as a container?

<comment><USER11> At the host level - yes. All the applications still make the same requests to the kernel. Different namespace doesn't matter.

</comment></comment><comment><USER18> Great article. I'd also recommend people take a look at ThreatStack for aggregating syscall events: [https://www.threatstack.com/](https://www.threatstack.com/).

</comment><comment><USER19> [duplicate of [https://news.ycombinator.com/item?id=13009796](https://news.ycombinator.com/item?id=13009796)]

<comment><USER20> True, and of [https://news.ycombinator.com/item?id=13007726](https://news.ycombinator.com/item?id=13007726) before it. But there seems to be interest, so we've moved the comments to the post currently ranked highest (that would be this one).

</comment></comment><comment><USER21> This is super great.

<comment><USER22> Yes it is</comment></comment>


Paul Graham's response to AOC's statement on billionaires - <AUTHOR> https://twitter.com/paulg/status/1292219835804528653 <comment><USER2> Startups would not have existed if not for free and open source software. You know, Stallman and the like, which people like Graham pretend don't exist or don't matter. But than again, a person that retweets trump is not someone I'm particularly interested in hearing what they have to say in general.

</comment><comment><USER3> While I really like the open debate directly between influential people without journalists gatekeeping, I don't think Twitter is a good medium for this.

There's only so much nuance you can convey in a few short sentences.

Really wish politicians wrote essays to persuade their case instead of just tweets or news interviews.

</comment><comment><USER4> Ridiculous. PG says that startups cause billionaires as though this is a law of the universe. Why should it be? It would surely be possible to structure society in such a way that startup founders of unicorns don't obtain more wealth than they can possibly spend in a lifetime.

Further, not all startups become decacorns or megacorps. Small business exists, even though PG and the broader VC community would rather focus on the startups that make it huge. A sustainable business that employs 30 people, even under existing social structures, won't make the owner stunningly wealthy. And yet these businesses provide jobs just as well as the next uber or airbnb.

<comment><USER5> > they can possibly spend in a lifetime

Doesn't seem as a good working schema, in my opinion. Surely its possible to know how to spend quite large amounts, like quadrillions of dollars, in a lifetime.

<comment><USER6> Maybe we should not enable what happens after you get more than, say, $50 million. After you've bought a few really nice houses and given your kids the best education money can buy, all you're really doing with that money is imposing your will on others without that kind of money, and often without their say. Lobbying politicians, setting up foundations, etc., is usually a rich person telling others what they should do, rather than enabling individuals to do what they think is best. Maybe this is ok if rich people were really just always super intelligent, but I think it's clear that that's not necessarily true (luck (and sometimes even a bit of ruthlessness and lack of compassion) is big in getting rich). And people who are experts in one field can be clueless in other fields without realizing it. Just give the excess money to individuals, on the basis of need, and we'll probably see it used more efficiently.

<comment><USER7> Maybe we should ban prime numbers above 50 million also...

Or stop fighting reality...

:)

<comment><USER6> Fighting reality? Socialism has already existed in the real world.

</comment></comment></comment></comment></comment><comment><USER8> Startups aren't made by billionaires, the workers at those startup aren't billionaires, the people who make the most money out of a startup are the billionaires who invest in the first place.

The people who design an iPhone are seldom millionaires, ditto for the people who write the software, the people who actually build them, and the people who buy them.

Startups may make billionaires, but that doesn't mean those startups haven't benefited primarily from no -billionaires.

More over the bulk of the money those billionaires have are either cash (so not contributing anything to anyone), or investment funds (eg not adding capital to new businesses, and not injecting money into the economy).

So the bulk of assets held by billionaires are only benefiting those billionaires, and then they aren't actually doing anything with it except hoarding.

Billionaires don't add value to the economy. Because one billionaire is never going to be spending more than 20 _thousand_ times more than an average American. So they aren't contributing proportional to the amount they benefit from the economy.

</comment><comment><USER9> "Paul Graham @paulg · 15h ... she typed into her iPhone, and then posted on Twitter.

(Lesson: You don't need billionaires per se, but you do need startups, and startups cause billionaires.)"

Twitter and all but "startups cause billionaires"? Really?

<comment><USER10> Also I was basically in the room for the birth of the modern consumer smartphone and I'm not a billionaire. Huh.</comment></comment>


New DigitalOcean Droplet pricing - <AUTHOR> https://www.digitalocean.com/pricing/ <comment><AUTHOR> It looks like you can resize droplets using the control panel or API already, and I'm test resizing one server now. Basically double the RAM for the same prices, plus a few new 'flexible' instances where you pay $15/month and choose from more CPU vs. less RAM.

<comment><USER2> Not only RAM, but also disk space if you will resize permanently.</comment></comment>


Google and Nasdaq Pursuing Nano-Second Precision in Network Time Protocol - <AUTHOR> https://www.nytimes.com/2018/06/29/technology/computer-networks-speed-nasdaq.html <comment><USER2> "So-called high frequency trading firms place trades in a fraction of a second, sometimes in a bet that they can move faster than bigger competitors."

First off: no. Big money plays in high frequency trading (roughly half of all trading activity), and the smaller traders without instantaneous access are the losers in this game.

Secondly, NASDAQ's obsession with precise global sequencing is A) misguided and B) effectively impossible to do right 100% of the time. Given this, I would argue that the appropriate thing to do is change the market requirements. And I'd argue that like this:

1) Temporally quantize the market. Orders come in on an open temporal window that is sufficiently long to account for global latency of non-pathological communication (sorry, tor users) and a bit of computation time. Everyone gets to swim in the same pool. Maybe one second, maybe more. Nobody gets to see the order book until it's resolved. Write-only. 2) Lock the book and fulfill orders from the set of satisfiable orders. If there just contention for a trade (there will always be some), fulfill the contentious trades randomly using random zeedig generated from a pre-announced salt and a hash of some or all of the order book for the window. 3) Return the results and the hashes of the order book, next salt, etc, for verifiability and prep. 4) Re-open the order window.

High frequency traders would hate this, because they wouldn't be able to pounce on quick movements, even without fronting slower traders.

It would, naturally, increase latency for trades by virtue of having to wait for market resolution. However, mere sequencing doesn't solve the problem of having to resolve and confirm trades (the speed of light is so cruel), so I'm left utterly unsold on the market-efficiency benefit of ultra-high order resolution. Wealthy high frequency traders want to use time to buy an advantage, and the liquidity support they provide to the markets is dubious, at best, since they pull the plug as soon as things get crazy.

Quantize the markets.

<comment><USER3> Most HFT shops are relatively small. HFT is all about latency and turn over. Big quant shops might have HFT elements but lean far more towards systematic/algo strategies that can be relatively high latency (still super low latency, but not HFT) because these are the only strategies that you can deploy serious var with. The guys crushing HFT are not huge hedge funds, and they are solving more engineering problems than building trading models.

Also, no, small traders don't lose. Retail traders et al get much tighter spreads, cheaper execution by routing to internalizers, etc. It's big institutionals with size to trade that get front run and have to worry about HFT killing their shortfall. On the institutional side, it's about lit venues preferring HFT w/ special orders types and thin top of book. On the retail side, the issue mostly comes down to direct feed vs. SIP/CQS thanks to NBBO that opened the door for latency arb courtesy of yet more regulation. Blame your regulators folks. This is why dark pools became a thing.

> and the liquidity support they provide to the markets is dubious, at best, > since they pull the plug as soon as things get crazy

This bit is certainly true.

Source: hedge fund trader who hates HFT not in principal but because they are good at what they do

<comment><USER4> Of course the retail guy gets hurt. Who do you think buys the mutual funds that pay a higher price.

Seriously, I hear this all the time, but it's only one step removed. Why does everybody keep repeating this lie?

<comment><USER5> Vanguard states that hft activities have lowered their trading costs, so I don't think this claim is necesarily true.

<comment><USER3> HFT has reduced costs under 99.99% of market environments. My direct cost and slippage is still so much lower than it would have been 30 years ago. Hell, even 10 years ago.

<comment><USER6> Where would the profits that go to HFT outfits go if HFT wasn't a thing? Genuine question, I have no idea how that works.

<comment><USER7> They would simply vaporize. At the margin, people trade because the frictional costs (spreads, fees, pricing/tracking error, risk) of trading are low. Fewer people would trade.

HFTs basically play an intermediary role: risking capital to buffer supply/demand imbalances, aiming to buy things at a discount or sell at a premium to their perceived value. The more transactions an intermediary does, the smaller his margins per transaction can be. Low margins fuel even more transactions in a virtuous cycle, and competition drives margins down.

Take this thought experiment to an extreme level. What would happen if short term speculation were banned, all stocks traded January 1, and had to be held for a year? Only very wealthy people with high risk tolerance could participate in the market, since they couldn't sell companies at will to fund personal expenses or if the business underperformed.

Volumes would plummet. Exchange/brokerage fees would be a significant percentage of the deal size, similar to what real estate agents charge, since they can only do a few transactions. Intermediaries would be something akin to a private equity fund, bidding 10-20%+ under value to cover the risk of holding for a year.

Even with trading reduced to once a minute/hour/day, many trades HFTs take the other side of now--say a medium frequency quant fund believes a company is underpriced by 0.1%--simply would not exist anymore, because spreads and fees would increase. Most ETFs would disappear. The marginal cost for an HFT to make markets in some small ETF is basically 0, but a human would make more at McDonalds than market making an ETF that trades a few hundred thousand shares a day.

<comment><USER8> As noted elsewhere in this thread, I suspect new markets would spring up. If the underlying could only trade once a year, the options market would be huge.

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> This looks like a paper about the Huygens system, with comparison to other protocols like PTP: [https://www.usenix.org/system/files/conference/nsdi18/nsdi18...](https://www.usenix.org/system/files/conference/nsdi18/nsdi18-geng.pdf)

Also these slides: [https://platformlab.stanford.edu/Seminar%20Talks/retreat-201...](https://platformlab.stanford.edu/Seminar%20Talks/retreat-2017/Yilong%20Geng.pdf)

<comment><USER10> I maintain an NTP implementation. That comparison doesn't seem fair to me. It looks like they are comparing the old reference NTP implementation and not really the protocol itself. An NTP implementation can certainly synchronize clocks with better accuracy than 1 millisecond, or even 1 microsecond with hardware timestamping and good network switches.

There are some interesting ideas in the Huygens paper, but I don't see anything that couldn't be done also with NTP.

<comment><USER9> The other thing I don't understand is that this paper argues against hardware timestamping on the grounds that users won't want to buy expensive hardware, and that Huygens is for "standard hardware ... in current data centers". Expensive, niche hardware is _normal_ for the HFT folks that care about nanosecond precision.

<comment><USER11> The paper did not have hft as its primary use case, but you cannot contain reporters tendency to cherry pick eye catching aspects.

<comment><USER12> Yes, it seems that the designers tried to make something that would work in the current Google data centers. Which is not surprising as some of the authors work at Google.

I'd more want to know: why does Google need even better synchronization of time stamps than what they have now?

</comment></comment></comment></comment></comment><comment><USER13> It is great news if this algorithm can work directly on the public Internet, without requiring a specialized network - many scientific and engineering applications will be able to get its time reference directly from the Internet!

NTP and other protocols currently used are unauthenticated (there is NTP autokey, etc, but its security properties are not ideal, and mostly not deployed) and it is a big security hole, especially more and more cryptographic programs are being putting online, since this protocol is meant for financial applications, hopefully the security issues can also be solved by using digital signature.

I guess it's a rare scenario which the finance industry makes a _direct_ contribution to technology.

<comment><USER14> I can't imagine how you could hope to reach nanosecond-precision over the internet without changing all the routing hardware in use today. With PTP you can already reach sub-microsecond synchronization but you need full hardware support for every network element on the route if you want to achieve that. The timestamping is done on the network PHY itself, which is how you can remove all jitter introduced by the kernel stack. Anything receiving and re- transmitting the packet along the way must update the timestamps to account for the processing delay.

Huygens is a little more clever and uses a statistical approach to sample multiple clocks and correlate them (if I understand correctly) however it still seems designed to work within a datacenter, I assume that over the internet the signal-to-noise ratio for measurements would worsen very significantly and lower the precision dramatically. It could well still outperform NTP however.

<comment><USER15> The signal-to-noise ratio would be a nuisance, naturally, but there's a more fundamental problem that's stopped anyone from trying this. Internet routing and latency is asymmetric - your request to the time server will, in general, go via a different route that takes a different amount of time compared to the message back. This introduces a bias that is indistinguishable from local clock error. Without precisely synchronised clocks at both ends, you can only determine the overall round-trip time and not the time each leg took, and therefore you cannot calculate the exact difference between your local clock and the server. This is an inhererent limit on how accurate NTP can be over the Internet. (The other reason not to bother is that GPS is cheap and can provide a very accurate synchronized clock source over long distances.)

</comment></comment></comment><comment><USER16> This technology will also be excellent for games. Right now, the server has to decide how it's going to break ties, and that results in confusing moments. For example, Overwatch favors the shooter, so you might see someone in a position to shoot you, use a defensive ability, and die even though you used it, because the server decides that if two events are close in time, the shooter wins. With a trusted time reference, you can actually order events from the perspective of the client instead of the server; so if you actually used that defensive ability before the shooter clicked the mouse, you don't die.

I actually think NTP might be good enough for this (despite what people say their reaction times are, milliseconds don't actually matter), but I guess game developers don't think of games as globally-distributed transaction processing systems (which is what they are, just a lot more write conflicts to resolve than your average database), and haven't experimented with ideas that are still only a few years old in that field. (The game industry also doesn't reward experimentation. If you're Google and you try to replace Bigtable with Spanner and it fails, it doesn't matter, you just keep using Bigtable. If you're a game company and your netcode is janky, you launch late, a competitor releases a similar game before you, and all the money you spent on development is gone.)

With games there is always the trust issue; can someone write a client that lies about the time they took an action? The answer is yes. But if we have technology that relies on similar client trust working in high-frequency trading, it should be safe enough for games. The stakes are a lot lower in a computer game than the financial markets. So I think good things are on their way.

<comment><USER17> You still cant trust the client here, no matter how precisely synced your clocks are, because it could maliciously reorder events/tamper with time (always claim you activated first, with a certain fudge factor to prevent detection) and it would be undetectable within the bounds of internet latency.

<comment><USER16> I'm not completely convinced that cheating would not be detectable after aggregating some statistics over time. (Similar to how you can extract encryption keys from server processes, simply by timing how long certain operations take.)

<comment><USER17> The problem is a statistical approach to anti cheat like that results in banning tons of innocent players, which is awful PR.

</comment></comment></comment></comment><comment><USER18> HFT aside, this also has interesting applications in distributed databases. Spanner, for instance, has consistency guarantees directly related to how synchronized the clocks are (keyword: TrueTime). It seems plausible that Huygens could make a significant improvement on performance of this kind of distributed database.

This looks like a very impressive result. NTP has been doing its thing well for years but a factor of 100 improvement on time accuracy would be amazing,

</comment><comment><USER19> IEEE 1588 v2 + rubidium decay grandmaster time sources do this quite nicely.

Source: have worked as a software / Linux monkey for two of the biggest US HFT firms the past 10 years.

</comment><comment><USER20> It's interesting, that time synchronization in a rotating reference frame is path dependent ([0], for example).

[0] [https://tycho.usno.navy.mil/ptti/1974papers/Vol%2006_26.pdf](https://tycho.usno.navy.mil/ptti/1974papers/Vol%2006_26.pdf)

</comment><comment><USER21> From [0]:

> In this paper, we present HUYGENS, a software clock synchronization system > that uses a synchronization network and leverages three key ideas. First, > coded probes identify and reject impure probe data—data captured by probes > which suffer queuing delays, random jitter, and NIC timestamp noise. Next, > HUYGENS processes the purified data with Support Vector Machines, a widely- > used and powerful classifier, to accurately estimate one-way propagation > times and achieve clock synchronization to within 100 nanoseconds. Finally, > HUYGENS exploits a natural network effect—the idea that a group of pair-wise > synchronized clocks must be transitively synchronized— to detect and correct > synchronization errors even further.

Not an expert, but this seems quite a complex system. Since HF traders have huge incentive to game the system, my fear is that the next headline about Huygens will be about a new exploit found in the wild.

[0] [https://www.usenix.org/conference/nsdi18/presentation/geng](https://www.usenix.org/conference/nsdi18/presentation/geng)

</comment><comment><USER22> How would you use that nano-second precision on a CPU/OS that supports at most microsecond precision?

<comment><USER23> Only if the article mentioned it!

Oh wait, it did!

[https://www.usenix.org/conference/nsdi18/presentation/geng](https://www.usenix.org/conference/nsdi18/presentation/geng)

<comment><USER9> I don't think the article answers amelius's question, which is not about how you would _achieve_ that precision but how you would _use_ it. On an OS that only supports recording timestamps up to microsecond accuracy, what is the point of synchronizing your clocks within nanoseconds of each other, even if you could?

<comment><USER24> On linux you can get nanosecond timestamps, and so could record them. This could be useful for capturing certain events. Is that what is meant? IIRC Windows XP didn't support resolving timestamps smaller than 8ms but we've moved beyond this.

</comment></comment></comment></comment><comment><USER25> Confusing title. Perhaps better to say "successor to Network Time Protocol", which, like IRIG or PTP, is a pretty common timekeeping protocol.

</comment><comment><USER26> Seems sad so much technical effort to be exerted towards something so venal, trivial, and fleeting.

<comment><USER27> If there is money in it, it will get solved!

</comment></comment><comment><USER28> I think this is no longer about NTP but all the stack below it including hardware.

</comment><comment><USER29> What value does this add to the world?

</comment><comment><USER30> I wonder if the next wave of hft will be a geostationary space station over New York, to get the extra occasional nanosecond.

<comment><USER2> Uh, I know it's a joke, but isn't geostationary orbit almost 36K km over the equator? The round trip time to NYC would be over a quarter of a second...

Hence, drones. With lasers... Laser drones...

</comment></comment><comment><USER31> An interesting side-effect of this, is that it would enable a standard of synchronization, across geographic regions, such that one could treat a set of virtual machines as one ultra-wide-bus CPU with a 1 GHz clock speed.

All of the local overhead of real system resouces and network synchronization could handled by the remainder of the real CPU clock available to the bare metal, but contribute to the computation of a segment of a virtual bit field, at speed.

So, now maybe we get a commodity 4096 bit 1 GHz CPU as a service. Which, is maybe comparable to a 64 core processor, but without the overhead of chunking down to the width of 64 bits.

<comment><USER32> Are you saying that 64 bit CPU + 64 bit CPU = 128 bit CPU (as long as they are time synced)?

1\. It doesn't work this way 2\. Why would you want a 4096 bit CPU?

<comment><USER33> For financial transactions, it would certainly allow for fast high-precision floating point math. Imagine IEEE 754 4096-bit floats. Not sure anyone would _actually_ use this, and you'd still have to standardize the rounding precision, but it might be an interesting vein of research.

Still, I agree with you -- what the OP described is _not_ a 4096-bit processor.

Now highly-synchronized VMs -- that's an entirely different matter. Probably a boatload of use cases for those.

<comment><USER34> Why would you use floating point math for finance?

<comment><USER35> A floating point representation is not really the issue, the issue is not using base 10, and IEEE 754 specifies base 2 and base 10 floating point formats and operations. But I am of course not sure whether the original comment referred to base 2 or base 10 and given how common the mistake of using base 2 floating point numbers for financial calculations is, you may be correct with the intention of your comment.

<comment><USER33> I'm aware of the fact that you don't use floating point math for finance -- for exactly the reason you described -- but the academic in me wonders if you could formally specify a high-enough degree of precision -- and all the corner cases -- to allow FP math for even just a subset of transactions. This would (in theory) allow to programmers to bypass the Decimal classes in your favorite OO language (or GMP if you're a C fan).

Again, purely an academic inquiry :-)

<comment><USER35> My point was more that it is wrong to say that financial calculations should not be done using floating point formats, for example Decimal in .NET and BigDecimal in Java are floating point formats and they are the types you should use for financial calculations. The important difference as compared to formats like IEEE 754 binary32 (formerly single) and binary64 (formerly double) is that the representation is based on base 10 instead of base 2. Fixed point or floating point and base 2 or base 10 are two orthogonal choices.

So when you initially mentioned high precision floating point numbers for financial calculations that was not necessarily a bad idea because you might have thought about base 10 floating point numbers. The comment I replied to however assumed you meant base 2 which of course most people do if they say floating point numbers without specifying the base and which of course is a bad idea for financial calculations more often than not. I just pointed out that assuming base 2 is usually but not technically correct.

And you can of course use base 2 floating point numbers for financial calculations - 32 bit, 64 bit, or 4096 bit - you just have to keep track of the accumulated errors and stop or correct the result before the error grows into the digits you are interested in. But why would one want to do this? The only thing I can really think of is that you need maximum performance and there is no hardware support for base 10 floating point numbers. And just using integers as base 10 fixed point numbers, which would often be a even better solution, must not be an option.

</comment></comment></comment></comment></comment></comment></comment><comment><USER36> This is a huge waste of effort. HFT should be banned so smart people work on solving real problems instead.

<comment><USER37> > _so smart people work on solving real problems instead_

Like making people click on ads?

</comment></comment><comment><USER38> Why not just switch to PTP (IEEE 1588)?</comment>


Shinken: A Pythonic Nagios - <AUTHOR> http://www.shinken-monitoring.org/features

<comment><USER3> I read the "Why do this project" section on the site, and am not convinced. Nagios is a very actively maintained project, and did have major features added about a year and a half ago. It is totally suited for use in large installations, especially with the help of existing third party enhancements like GroundWork Monitor. And on the non-issue of wanting a flexible Nagios, a lack of flexibility is NOT one of Nagios' problems. It is insanely flexible, and has tons of add-ons and plugins. Also being in C, it performs very well, especially the current version. I've looked at the code, and it's not bad - maybe the shinken authors are just intimidated by C.

</comment><comment><USER4> Is there anyone out there who uses Nagios (and by use, I mean have more than a hundred services being actively monitored) that finds nagios lacking? Nagios is basically an engine for state tracking, notification, and cmd dispatch. Ironically, the base set of Nagios packages isn't capable of doing any monitoring - you need to add plugins to fill out the command library to do that.

Nagios, to me, is like cvs, jira, cacti, sendmail, bind - just one of those sysadmin toolkit apps that works pretty well and I've never really been inspired to really invest the time in learning the alternatives - never really needed them. (Yes - I know git/subversion blow cvs away, and postfix is probably the right MTA to use. I think most bind actually is the only one on my list that actually still has mindshare. Issue tracking (jira) is so over the place that I'm guessing there is no "one popular" system out there)

</comment><comment><USER5> We've been using a different nagios alternative, zabbix, for quite a while and I'm always surprised how few people have heard of it. The UI takes getting used to, but it's full featured and dependable.

<comment><USER6> happy zabbix user here, it's true, the ui is not especially obvious, but it works quite well, and with the graphing capabilities it is also a reasonable replacement for cacti.

</comment></comment><comment><USER7> Does anyone have experience with running large scale deployments of this? I am not a fan of nagios itself (last I looked at the code it was a horrible perl mess), but the idea itself is great.

<comment><USER8> Not very likely, as the project itself got wiki and ML only in January... But the architecture seems better than Nagios tbh. More distributed and with more options for redundancy.

Also the roadmap doesn't look great so far: <http://sourceforge.net/apps/trac/shinken/report/2>

But if someone started writing a drop-in Nagios replacement, I hope they know what the problems are and how to solve them.

</comment></comment><comment><USER9> Isn't that the same thing zenoss is trying to do?

/not really familiar w/ zenoss but I know their founder spoke at pycon a couple years ago

</comment><comment><USER10> interesting. i'm working on erlmon, an Erlang and Lua driven monitoring tool, and there are some good ideas in this tool that we may borrow.

<comment><USER11> Sounds cool. How far along are you? Have anything online yet?

<comment><USER10> up on github now. it works but we're still working out config file format, and it's just now able to send alerts. lot of work to do before we could even call it alpha...

<http://github.com/darrikmazey/erlmon/tree/master/erlmon-1.0/>

</comment></comment></comment><comment><USER12> This looks promising, especially since you can use existing nagios-plugins.

Anything that improves Nagios configuration is a good thing.

<comment><USER4> I've always thought that Nagios configuration was pretty good. For the small organizations, it's simple enough with it's object/template infrastructure, particularly now that you can create custom properties and pass arguments to nrpe, that most sysadmins can get a monitored environment up in 30 minutes, and then add devices on the order or 2-3 minutes per system (inheriting all the monitors for that object class)

When organizations get larger, the configs are straightforward enough that the machine generation (which is absolutely necessary when your monitoring 10s of thousands of services) is also relatively straightforward.

My company is dropping north of $1mm on ITIL practices, ticketing, trending, and KPI generators to depart from our current Jira/Cacti/Nagios world, but Nagios is one of the products that we're continuing to use for the foreseeable future. (Eventually that will change - I've discovered that Billion Dollar corporations would rather spend $500K / framework and then start adding developers (at $150K fully loaded) to extend that platform with proprietary in-house developed systems, rather than simply go buy an open-source product for $5K and do the same thing.

I'm not bitter though, really.

<comment><USER12> It may just be that the Nagios developers have a different cognitive style than I do, because I find the configuration language to be irritating to the extreme. So much so that I end up using templates to script configuration as much as possible. Somehow monit has always been more comfortable to use in my hand.

<comment><USER4> Interesting. About the only "Nagios Annoyance" that I have (and I don't really see a good way around it) is that you don't specify the commands you are running for a service check, but have to reference your command library (which in turn has the name and calling order of the actual parameters)

I realize that there is no way to update the command/parameters you are calling in one place without doing this, but the cmd entry usually almost _completely_ maps to the command that I'm calling - parameters, command name, etc.. that it's addition has never really seemed to be that critical - <USER2> always results in one more "lookup" when trying to figure out what a service check is doing.</comment></comment></comment></comment>