Show HN: Eno – A lightning fast, user-friendly YAML/TOML alternative - <AUTHOR> eno [1] - A modern plaintext language w/ libraries [2] for JavaScript, Python, Ruby & soon more!
We migrated a big relational research database to a file-based solution - requirements were:
- Super fast and easy editability for users
- Highest performance for parsing/validating >10K documents on every user change.
Our trials with YAML/TOML showed us that we wanted something both faster [3][4] and easier [4], something tailored for file-based content management ... and after months of research & development it's now publicly available (under MIT license) for everyone!
Last but not least I also want to mention eno's document introspection capabilities - with a few lines of code you can build intelligent relational suggestion UIs as shown in [4] below.
[1] https://eno-lang.org/
[2] https://eno-lang.org/libraries/
[3] https://github.com/eno-lang/benchmarks/
[4] https://eno-lang.org/resources/introspection.mp4
PS. Your input for the Roadmap is highly welcome - what do you think should be in the next releases? More languages? (If so, which? Currently in progress: Rust/PHP, Currently planned: Go/Java) Additional IDE/editor support? (Currently supported: Atom/VSCode/Sublime) Or something else entirely? :) Looking forward to your feedback! <comment><USER2> After a quick look I have a lot of questions

1) Unlike YAML or JSON this doesn't parse into a simple array structure, but a library dependent object hierarchy?

2) Is the API of the libraries also part of the language spec?

3) Can I assume that document.lookup() will be available as document->lookup() in the PHP library?

4) What about programming language specifics that may differ between languages? Will the PHP objects implement the Iterable interface? Or the ArrayAccess interface? (There are probably similar but slightly different concepts in other languages).

5) There is eno.parse(), but is there some kind of reverse mechanism to create a new Eno document? Like document.addList([...]).addSection('hey', 2) or something.

<comment><AUTHOR> 1) Yes! (You can directly dump it to a language-native structure with the raw() method too, this is not 1:1 YAML/TOML style generic deserialization though as there are no fixed types in eno)

2) Some detail aspects of whitespace-parsing around the line continuation syntax will need to be specified by the language, the shared official API I am implementing for the different platforms is fully open to improvement and future reinvention though, I'd love to see a completely new take for a library API if it comes up in the future. :)

3) Definitely!

4) I try to keep things as consistent as possible across the platforms, but if there are important language specific paradigms I think these should be taken advantage of! I can't answer details regarding the PHP implementation yet but keep in touch, I'm happy about a dialogue here! (Also I can't be good at everything :)).

5) I want one! Obviously there can't be a stable generic "just dump it already" implementation, but a smart builder-type API is definitely on the list, I even started one for enojs but had to re-prioritize because there was so much else to do for the whole ecosystem. ;)

</comment></comment><comment><USER3> What I was looking for on the website and think is more than implementing a parser in another language is schema support. That is, you should provide something like XSD for XML, JSONSchema for JSON, TOLS for TOML.

Why? There is a need (see above enumeration) to declaratively specify how an Eno file should look like. I do not want validation to creep into my code, like you do with `document.string('author', required: true)`. This just scares the hell out of me. Say you want to parse some Eno file with different languages, you also end up replicating validation, which mains you end up maintaining it, or rather not maintaining it... Apply leverage by moving validation into your parser.

Another thing is that it appears you are implementing the parsers by hand instead of using a parser generator that consumes a grammar for Eno. What is your reasoning behind this? Is it performance? Did you benchmark using generated parsers (maybe wrapped in a nice API)?

<comment><AUTHOR> Yay thanks for the detailed input!

If there is demand or initiative for a portable schema solution I'll gladly support it! The native architecture in the eno libraries is programmatic because that does have its own powerful merits which are employed to the fullest in the API design, like always there's not one best choice, and the 'validation creeping into code' can be turned around into 'external schema definition creeping out of line with code' just as well. ;) Do you have some concrete usecase in mind or planned where we could explore how a portable schema solution could look like for eno? Drafting things from various real life use cases has worked great for eno so far, so that's the route I would love to go here too if we follow that track!

Custom parser implementation is easier to answer: By now I've iterated through dozens of custom parser designs for eno in multiple languages, and I'm pretty much confident that generated parsers will not stand a chance of being faster, they after all do the same thing as I do, only I can't really hand-optimize what they produce afterwards. :) You can study the benchmarks I linked to under [3], there are some generated TOML parsers included with rather disappointing performance to put it mildly, and as it stands there's not much that's faster than the eno parsers in YAML/TOML land anyway, so I have low incentive to experiment in that domain currently. :) Long term goal is to (optionally) integrate (generated or custom) C (respectively Rust) parser cores through native bindings as well, so that will bring up this question again then for sure.

<comment><USER4> I've come to really appreciate the difference between "syntactically correct" (ie: is a file valid xml) vs "semantically correct" (ie: does it follow a specific dtd if it's valid xml.) More than that, I've come to realize how many other people don't have this appreciation even though they will identify problems that directly relate to this distinction in every day usage.

To truly be able to have a portable file format, there needs to be a way to do both validations reliably in different contexts (eg: different languages). If you ignore this part of your design then it may become the slowest part of the eno ecosystem because your grammar _will_ have quirks that you'll end up needing to support long-term. I suggest toying with this functionality now and providing something which is extremely pessimistic on what it will pass. Only loosen things up as people show a need and keep your entire spec as tight as possible.

I would imagine that you could even use eno syntax to describe document structure, much like xml/dtd has such strong parallels with each other. Then you get the fast parser in both places essentially for free!

Finally, on the format of eno itself, I'm curious on your thoughts relating to unicode characters that visually masquerade as common characters. eg:

[http://www.fileformat.info/info/unicode/char/ff1a/index.htm](http://www.fileformat.info/info/unicode/char/ff1a/index.htm)

Sample usage:

\---

author: Jane Doe

email: email@example.com

\---

Does this parse?

How about this:

\---

author: Jane Doe:

email: email@example.com

\---

What do I do if I want a "#" in a name?

\---

# #twitter

@hackernews = 0xC0FFEE

\---

or:

\---

# \\#twitter

@hackernews = 0xC0FFEE

\---

Are quotes optional somehow? Can I put arbitrary things into an identifier?

Cheers and keep up the great work!

<comment><AUTHOR> Syntax vs Semantics is distinguished by _ParseError_ vs _ValidationError_ in the eno libraries - I'll keep the importance of distinguishing them in mind for the schema development too - thanks for pointing this out!

Right now only an ASCII colon is interpreted as an operator, but this looks like a question to thoroughly consider for the next and final spec (which is planned for 2019, currently we're in frozen RC) - work on this currently happens at [https://github.com/eno-lang/eno](https://github.com/eno-lang/eno).

There is escaping for arbitrary keys by using backticks - see the advanced language feature documentation at [https://eno- lang.org/advanced/](https://eno-lang.org/advanced/), in the case of # #twitter you wouldn't need it though unless you omit the space.

Thanks for your input, appreciate it!

</comment></comment></comment></comment><comment><USER5> What about types?

All parse to string? If not, hopefully please add date/datetime support (better as ISO) and decimal.

\---- > More languages?

How about make the core on Rust and the rest use it?

BTW: What your use for the introspection? What editor is that? I like the auto-complete stuff..

<comment><USER6> This! As the introduction doc coyly says,

> so as a user we usually just concern ourselves with editing the values and a > friendly developer takes care of specifying the names for us. :)

The friendly developer gets the job of conveying -- somehow -- that values for landline: and mobile: have to be valid tel#s (but must they have the country prefix?) while "hire date:" must be in the form yyyy mm dd only... and also gets the friendly job of writing one-off, locally unique validation code for these fields and hacking them into the parser.

<comment><AUTHOR> Conveying what types to enter is not an eno-specific problem, as a user without schema or code access you don't know which types a blank YAML/TOML file expects either!

Asides the absolutely valid meta solutions (e.g. in-file comments, clear key naming, documentation) there is an additional way this is approached in eno: If you use the type loaders provided by the API (say 'my_var = document.url('website')'), and properly expose errors to the user, the user will get a localized (!) error message in his language, like "'website' must be a valid url (e.g. [https://eno-lang.org/)"](https://eno-lang.org/\)").

In the long run we can have community packages for any number of important locally unique types (loaders are just simple functions, so they can be easily authored), so at some point you likely don't have to write any one-off validation code, and neither the error messages or their localizations, you just pull it in as dependencies.

<comment><USER5> How about type inference? You can look at rebol/red/tcl for inspiration, that already look like a config format but have a defined way to types:

[https://randomgeekery.org/2004/12/26/rebol- datatypes/](https://randomgeekery.org/2004/12/26/rebol-datatypes/)

[http://www.re-bol.com/rebol.html](http://www.re-bol.com/rebol.html)

<comment><AUTHOR> I appreciate the input :) but the thing is that the typing concept in eno as it is now is essentially what makes eno eno. Every application that uses eno decides for itself what types it supports and requires, and that in turn is how eno manages to be so simple and usable on the language level, even for completely non-technical people who normally feel uncomfortable with the idea of editing their content as raw text files.

If I would add types and type inference again, then I would essentially arrive at YAML and TOML again, and I don't want to reinvent them. ;)

But if I actually misunderstood you there, please let me know and do clarify!

<comment><USER5> I get that, but I wonder how give a base set of types, that avoid small incoherences.

JSOn is a good example:

[https://www.tutorialspoint.com/json/json_data_types.htm](https://www.tutorialspoint.com/json/json_data_types.htm)

Is so spartan that everyone need to encode dates somehow, to make a simple example. I think this are the base types (also, my experience with RDBMs and building a relational lang now, and always having troubles with cvs, json, and others formats in ETL kind-of-task):

\- String

\- Floats. Can be split Ints/Floats but stick to just Float is ok. However, make it Float64.

\- Date(Time). And be ISO. Not ambiguity.

\- Boolean

\- Decimal64. This is a pet peeve of mine. A lot of data in the business space is about money, and floats are not ok. What if like in rebool $3.2 is decimal?

Then the composites.

ie: This is json + dates/decimal. And make a single encoding (utf8?).

Is insane that, for example, you save a CVS in excel and open it again and excel get lost, and can't parse it fine.

Apart from this, url, email, host, website, phone, cellphone, city, country, state are so common that maybe with a import like "!schema:common-fields" or something.

</comment></comment></comment></comment></comment></comment><comment><USER7> It seems bizarre that you wouldn't create a C version of this library. If you create a C version, everyone can write bindings and consume it from their favorite language. I think maybe you can do that with Rust code, but it's typical to write the canonical version in C.

Your benchmarks are flawed, as they only compare between different implementations using the same language. If you really care about performance, it seems bizarre that you'd use PHP or Ruby.

<comment><AUTHOR> That road (C or Rust parsing core through bindings) will likely be taken, but for the initial development and jump-starting the ecosystem it was important for me to start with implementations that can be quickly experimented with and iterated and not spend a lot of extra time on dealing with segfaults, memory leaks, the different binding mechanisms on different platforms, etc. As things stand now, people are provided with multiple, fully functioning, pure implementations that already are faster than the majority of YAML/TOML parsers. In the coming months and years there will be plenty of time to make things even faster. :)

For me caring about performance also means caring about performance on all platforms, why not after all? You can take the tabular benchmark data I provide and paste it together, or use the raw data that is also available as eno files in the repository to compare language against language too (which I initially also did but later dropped because same-language comparison for libraries made more sense to me), if you want the quick run down as far as I remember it: mostly javascript parsers lead the ranking, ruby parsers are a bit behind and just slightly ahead of Python.

</comment></comment><comment><USER8> The level of sophistication you got this to over the last few months is truly impressive. Supporting several languages and IDE/editors from the start takes serious dedication for one person. Kudos! Looking forward to using it for my next database-less project.

<comment><AUTHOR> Thanks michael! <3

</comment></comment><comment><USER9> I'm left looking for a spec, like a real spec-flavored specification with all the gritty details, written like you would want if you were writing a parser. It doesn't seem like there is one, yet.

<comment><AUTHOR> You're right, _not yet_! Jump-starting the whole ecosystem was a major time investment for me but now that there is public exposure providing a formal spec has a higher priority because someone might actually see it and do something with it ;) Keep an eye on [https://github.com/eno- lang/eno](https://github.com/eno-lang/eno), this is where I'm working on it, I'll also announce it on the newsletter ([http://eepurl.com/dA9LcH](http://eepurl.com/dA9LcH)) when it's there!

</comment></comment><comment><USER10> Related as a better config language (but universal, not for a use case like this) is dhall (non-turing complete, type-safe, remote imports). [https://github.com/dhall-lang/dhall-lang](https://github.com/dhall- lang/dhall-lang)

<comment><USER11> At first glance dhall looks a bit like some of the CSS preprocessors out there.

</comment></comment><comment><USER12> first off, I agree that file-based content management is the way to go. thank you for working to make better tools for it.

> we wanted something 1) faster and 2) easier

1) a) why didn't you write a new library using the same spec? b) do you have speed tests to show your libraries are better than existing ones?

2) how is this spec easier? (like a basic rundown)

<comment><USER13> > b) do you have speed tests to show your libraries are better than existing > ones?

The OP linked to their benchmarks in [3]: [https://github.com/eno- lang/benchmarks/](https://github.com/eno-lang/benchmarks/)

</comment></comment><comment><USER14> Honestly, putting a pair of parentheses in all of these plain text format, you get back lisp.

Greenspun's tenth rule rules!

</comment><comment><USER15> Can you talk about why "" We migrated a big relational research database to a file-based solution""?

I'm curious how modern DBs failed you here, more details of the problem space please.

<comment><AUTHOR> Sure!

Cultural research = notoriously underfunded, so although they have and rely on a relational database that holds their data (previously Postgres) the cost and effort associated with maintaining and extending the system is pretty high.

With the new setup the thousands of eno files represent both the place of storage and the interface to edit the data, so by that we eliminated the development effort to provide and maintain a full web frontend to the database, and the effort to just maintain the actively deployed technology somewhere and keep it at least patched for security reasons.

All that remains now technology wise is an Atom plugin that is locally installed on each client at the institute and takes care of validating, provides relational autocomplete helpers as demonstrated in [4] and offers a few hooks to kick off local builds for multiple deployment targets and deploy them to live as well.

Hope this clarifies things! :)

</comment></comment><comment><USER16> Kinda worthless rant, but I'm tired of reconsidering configuration file markup language every couple of years, and please remind me: why TOML isn't perfect?

<comment><USER17> If you already believe TOML is perfect, why do you reconsider configuration file markup languages every couple of years?

</comment></comment><comment><USER18> It looks like there is no support for writing data to this formatting. My inner hope was that there was a format I could parse, modify, and write again preserving as much comments and formatting as is reasonable.

I looked at the Python implementation but did not see that type of functionality. Am I wrong?

<comment><AUTHOR> ruamel.yaml in python does that to a certain degree from what I've read, you might want to check it out if yaml is ok for your usecase! ([https://yaml.readthedocs.io/en/latest/](https://yaml.readthedocs.io/en/latest/))

I've given this some thought as well, and given that the eno libraries hold their own representation of data in memory this might actually be plausible to implement in some way. Still I fear this will turn out to be a hard, hard problem (as eno is not even generically serializable by design), so that's why I haven't explored it further. So for the moment I can only say - Maybe in the near future sometime, check back every once in a while! :)

</comment></comment><comment><USER19> I like this, but I'm wondering how strict the format is. It seems like a potentially useful format for capturing data from a non- or semi-technical user base, but then some degree of fault tolerance in the entered data would probably be desirable. Was this (or should it be?) one of the design goals?

<comment><AUTHOR> One of the design conderations was and is that the format is very strict (and that way predictable), but at the same time as helpful as possible in identifying, communicating and resolving issues.

To that end all error messages that can occur are handwritten, fully localized and shared across all eno libraries (see [https://github.com/eno-lang/eno- locales/blob/master/specific...](https://github.com/eno-lang/eno- locales/blob/master/specification.eno)) and the API implicitly handles them for you when you write programs that consume eno.

So basically eno does no magic fallbacks of any sort when faults occur, but it is candid and friendly about it when it happens. :)

</comment></comment><comment><USER20> I just read the benchmarks and was surprised at how slow the toml parsers are, I mean it's syntax is simpler to parse than YAML (assuming spec v1.2) but they where still slower...

So I guess there is simply no high performance toml parser implementations?? (in the tested languages, which don't include rust)

<comment><AUTHOR> From what I saw, I think for at least a few parsers this might be the case because they are built on generated parser code, and it's just easy to run into unfavorable bits and pieces in the output that way, which can drag down performance completely although 95% of the parser are just fine. Technically there's no reasons why toml parsers shouldn't be just as fast or faster as yaml or even eno. :) In any case I'd be happy if the benchmarks stir up some movement and maybe kick off some high performance toml parser intiative, toml is an awesome format and also 0.5.0 was just officially released so there's a good reason to update the parsers now anyway. :)

</comment></comment><comment><USER21> I am working on a spectrum diagram rendering system and have been thinking hard about what syntax I should select for source documents. I'm going to give eno a hard look and the existence of JavaScript parser is not the least of my reasons. Thank you.

<comment><AUTHOR> That's fantastic to hear! Do let me know via email or github etc. if you run into any issues, I'm eager to gather more insights from other people's usecases and work on improvements where needed!

</comment></comment><comment><USER22> This is amazing. We really needed an alternative to YAML, and this seems great.

<comment><USER23> Have you seen strictyaml?

<comment><USER22> No, but I'll look it up. The name is not bad.

</comment></comment></comment><comment><USER24> Is there support for dictionaries as list items / nested dictionaries?

<comment><AUTHOR> Yes through _sections_! See [https://eno-lang.org/introduction/](https://eno- lang.org/introduction/).

You can nest as deeply as you want and multiple sections on the same level automatically turn into a list of sections. For just a list of flat dictionaries you can also use fieldsets, see [https://eno- lang.org/advanced/](https://eno-lang.org/advanced/). :)

</comment></comment><comment><USER25> good name, good possible alternative to yaml (I juste hate json), and not the least a ruby gem ! I will watch for the project, on how to evolves ....

</comment><comment><USER26> How does it handle Oblique Strategies?

<comment><AUTHOR> On npm that's in fact already covered, scrutinize the list [https://www.npmjs.com/search?q=eno](https://www.npmjs.com/search?q=eno) :)

</comment></comment><comment><USER27> Thanks, but I would keep using SDLang : [https://sdlang.org/](https://sdlang.org/)

</comment><comment><USER28> Do you remember when people were writing plain C parsers first? Pepperidge farm remembers!

</comment><comment><USER29> Why would I use a field set instead of a section? They seem functionally identical, is it meant to be a semantic decision?

<comment><AUTHOR> Well spotted, good question!

It's also been asked in another thread on HN, I'm quoting myself here: "eno has neither indentation nor closing tags of any sort, that means if you use a section to group some values, you need to start another section to end the previous one (no closing tags!), that's why there are fieldsets, which allow short groupings that automatically end with the next field/list/fieldset."

Hope this explains it :)

</comment></comment><comment><USER30> no python 2 support means no python support

<comment><USER31> Python 2's end of life is in 1 year, 4 months. [https://pythonclock.org/](https://pythonclock.org/)

<comment><USER30> This is a meaningless deadline, given that nobody is actually paid to support Python. Third party vendors will support it for years to come.</comment></comment></comment>


Updates on Download.Com caught adding malware to Nmap installer - <AUTHOR> http://seclists.org/nmap-hackers/2011/6

<comment><USER2> at end of this "update" it asks to circulate the page, which includes the update. That page is already on HN</comment>


Internet Explorer, The Anime - <AUTHOR> http://www.youtube.com/watch?v=BHTUlF7NA2o

<comment><USER2> I know MS Japan and Taiwan is making some other tan to increase selling. Idk if other tech company should follow Microsoft decision, since Asia culture is a little bit different.</comment>


ZeroDB white paper: An end-to-end encrypted database - <AUTHOR> http://arxiv.org/abs/1602.07168 <comment><USER2> Interesting concept. I disagree with this statement from the "Data confidentiality" section:

> Unlike CipherBase [6], our approach doesn't leak information about the order > of the elements. By observing access patterns, the server can only figure > out which buckets are leaf nodes and which are branch nodes, but not whether > the child node is located to the left or right of the parent node.

The paper correctly notes that when a client does a range query, it leaks information about the total number of blocks returned. However, if I've understood correctly, it _also_ leaks the identities of those blocks. The closer two tree nodes are in order, the more likely they are to be touched by the same request. A passive adversary could use this to extract information about the relative ordering of encrypted nodes.

Also, active adversaries cause much worse problems. Let's say I have the ability to cause someone to do queries on the encrypted database, without observing the results. By simply doing separate index lookups on two different fields, I can extract a lot of information by looking at which nodes were touched by both queries.

Finally, I noticed that the paper doesn't describe how the B-trees are modified. If the client is the only party that knows how to decrypt nodes, then it's responsible for splitting and merging nodes to keep the tree balanced. Doesn't this require some kind of distributed coordination, so that multiple clients don't trample on each other's changes?

<comment><USER3> Also you cannot do queries on the encrypted db without being able to observe the results. E.g. if you are able to initiate a query on a fraction of the data, you already have access to that fraction of the data. And you cannot initiate a meaningful query if you are a server-side observer.

Of course, active adversaries could give wrong results (wrong pieces of the index) back, we didn't quite consider yet which problems that could cause.

<comment><USER4> Is there a reason that you did not cite Enigma in your list of notable works? They solve the adversary problem by splitting tasks among a network of nodes that post their intermediate work to a blockchain where it can be verified for accuracy:

 _Very recently, Baum et al. developed a publicly auditable secure MPC system that ensures correctness, even when all computing nodes are covertly malicious, or all but a single node are actively malicious [18]. Their state- of-the-art results are based on a variation of SPDZ (pronounced speedz) [19] and depend on a public append-only bulletin board, which stores the trail of each computation. This allows any auditing party to check the output is correct by comparing it to the public ledger's trail of proofs. Our system uses the blockchain as the bulletin board, thus our overall security is reduced to that of the hosting blockchain._

I'm not a db or crypto expert, but this seems like a promising solution to the active adversary problem if you have an honest, healthy blockchain that is incentivized correctly.

<comment><USER3> Yeah, actually that is true. Enigma worths mentioning as one of MPC references.

Looks like Enigma would have challenges in many practical environments (mainly because it involves a lot of small interactions between nodes, so should be affected by latency). So, it's more like a way to build "Etherium of private computations".

But, as you pointed out, it can have good clues how to solve the problem of active adversaries!

</comment></comment></comment></comment><comment><USER5> This looks very interesting.

I guess that this would be the next-best thing after FHE?.

The catch is the performance, right? I mean, the paper says:

"In order to make the database end-to-end encrypted yet still capable of performing queries, the client encrypts the buckets (at the time of creation or modification). The server, which stores the buckets, never knows the encryption key used. The objects referenced by the leaf nodes of the B-Tree indexes are also encrypted client-side. As a result, the server doesn't know how individual objects are organized within the B-Tree or whether they belong to an index at all. Since ZeroDB is encryption agnostic, probabilistic encryption can ensure that the server cannot even compare objects for equality. _When a client performs a query, it asks the server to return buckets of the tree as it traverses the index remotely and incrementally_ , as in Fig. 2b. The client fetches and decrypts buckets in order to figure out which buckets in the next level of the tree to fetch next. Frequently accessed buckets can be cached client-side so that subsequent queries do not make unnecessary network calls."

Which would suggest that it might be a bit slow to do many round-trips for each query.

I'm guessing there could be some specific use-cases where this is not so relevant?

I would love to have some more knowledgeable people comment on this as it would be really neat to be able to start using a DB with these features.

<comment><USER6> > _When a client performs a query, it asks the server to return buckets of the > tree as it traverses the index remotely and incrementally, as in Fig. 2b. > The client fetches and decrypts buckets in order to figure out which buckets > in the next level of the tree to fetch next. Frequently accessed buckets can > be cached client-side so that subsequent queries do not make unnecessary > network calls. "_

>Which would suggest that it might be a bit slow to do many round-trips for each query.

I was curious and found some information through a search (key point italicized) [1]: "On the performance aspect, with a real world use case of 1GB index, just 150KB of data must be transferred on average over three requests to fetch the results back. In full text search terms, 250MB of data can be queried in around 500msec which even though slow, may not be prohibitively slow for some use cases. Insert queries also may take around the same time. _The number of requests needed to fetch the query results grows logarithmically with the data size._ "

[1]: [http://www.infoq.com/news/2015/04/Encrypt-Database- CryptDB-Z...](http://www.infoq.com/news/2015/04/Encrypt-Database-CryptDB- ZeroDB)

>I'm guessing there could be some specific use-cases where this is not so relevant?

They claim the performance degradation is not much, but that would become worse as the number of clients grows. The clients also would need to have a relatively decent amount of RAM and CPU power to handle this.

Related: CryptDB from MIT - [http://css.csail.mit.edu/cryptdb/](http://css.csail.mit.edu/cryptdb/)

<comment><USER3> Yep, relevant pieces, thanks for bringing this!

Our performance become a little better since then. And also we'll update our fulltext search algorithm using Lucene's practical scoring function: that makes it more scalable and will also help with performance.

As number of clients grows, things actually become better for read queries (as long as we support MVCC). However, multiple writing clients can create congestion if they write data to the same place which can affect the performance indeed.

<comment><USER5> Do you have a specific use case in mind for this?

I ask because apparently there would be a practical limit on the size of the DB.

So considering that you would probably need to restrict the DB use to something specific, what would that be? (at least while performance gets better/practical for very big datasets)

<comment><USER3> I don't think it's so much of a size issue. I'd use it for applications where you don't have too many simultaneous users of the same dataset (in future - that constraint would be only about simultaneous writing users).

When users have their own private datasets, that's ok to scale up number of users though.

This limitation comes from a) invalidation requests and b) limited ability to do server-side conflict resolution

</comment></comment></comment></comment></comment><comment><USER7> Seems that this initial release only supports python. The client is obviously heavy, since it has to run all the queries, etc. Thankfully the protocol is well defined, so it's just a question of effort to support more languages.

[https://docs.zerodb.io/](https://docs.zerodb.io/)

<comment><USER8> They are our friends and i continuously asking them to make open standard instead of implementing everything on python and python database and make it work with psql or cassandra. I hope they will release something for this as we are wish to include zerodb to our encrypted messaging (actor.im).

<comment><USER3> In fact, the approach you used for actor (using GWT) is no less applicable to us (using pyjs or rapydscript - see [https://medium.com/@ZeroDB_/zerodb-js- zerodb-in-the-browser-...](https://medium.com/@ZeroDB_/zerodb-js-zerodb-in- the-browser-ffe013dca6f0#.fjtonf6u2))

<comment><USER8> Yes, i read it. But i am not talking about browser. What about IoT? What about mobile? Running python in mobile? No one will do this as it is just non- professional for majority of mobile devs. Some niche mobile apps on python is cool, but you will never find python iOS developer and you will never bundle python to your app.

Protocol and some clean reference implementation without dependencies will be neat.

<comment><USER3> Actually, I think it's not too hard to get a python library to iOS/Android (especially Android). But I see your point.

Reference implementation without dependencies would be just a more cross- language rewrite of these dependencies though, so would take quite a while. Good news is that authors of ZODB which we use as a dependency are willing to do that (and already do that) in a more cross-language way.

</comment></comment></comment></comment></comment><comment><USER9> I read the whitepaper, and while I have many questions one detail in particular jumped out at me:

"The client authenticates with an X.509 certificate or a self-signed certificate where the private key is derived from a passphrase. When a passphrase is used, the scrypt algorithm is used to derive a private key from the passphrase. Information about the salt is stored server-side and is given before establishing an authenticated encrypted connection. ... The symmetric key is derived from either the passphrase, the private key in an X.509 certificate, or provided by a key management service (KMS)."

You're deriving a private key from a user-defined passphrase, then storing the salt used to derive the password _on the server_ , which is defined to be untrusted. I understand you use scrypt to derive the private key, but this still seems like a massive, massive footgun. Database systems are usually kept up and running for a long time, like maybe even years. If the server gets the salt, isn't it only a matter of time before even a very slow brute-force attack can re-derive the authentication credential and make whatever queries it wants?

<comment><USER3> Deriving from passphrase is optional. Mainly, for development and testing, and also for users who are ok with passphrases.

Having a passphrase + server-saved salt makes it better than a passphrase with no salt (for brute-force by dictionary).

If you think about it, security of passphrase approach is tested all the time with Bitcoin brain wallets. Over there, private keys are derived using SHA256 (which is fast to compute!) with no salt. It's hard for humans to generate good passphrases, so these wallets do that for them (usually 12 words).

But yeah, in most production usecases it would be some certificate, or KMS

<comment><USER9> Yeah, this approach has been tested and has failed miserably: [http://eprint.iacr.org/2016/103.pdf](http://eprint.iacr.org/2016/103.pdf)

</comment></comment></comment><comment><USER10> I can't help but compare this system to Datomic. Effectively, what ZeroDB is calling the "database" is what Datomic calls a "storage service"; and what ZeroDB is calling the "client", Datomic calls the "transactor."

In both architectures, "blocks" of data get pulled from the storage service (database) to the transactor (client), which decodes blocks into rows of tables or indexes, does a query which possibly mutates those tables/indexes, generates new blocks from the mutated tables/indexes, and uploads them.

The difference is that ZeroDB's blocks are encrypted by the client, obviously—but I'm not sure that this is a _semantic_ difference in the architectures of the two databases. I'm wondering if an equivalent change could be made to Datomic's transactor logic, without materially affecting the way Datomic works. (The question is basically how much of Datomic's required block _metadata_ , necessary for finding the blocks on the storage service, leaks information which ZeroDB has managed to sequester inside the encrypted blocks instead.)

<comment><USER3> Huh, interesting comparison. Thanks for pointing that out! We'll read more about Datomic: maybe we can learn something useful from them

</comment></comment><comment><AUTHOR> BTW, you can hop in our Slack [[http://slack.zerodb.io/](http://slack.zerodb.io/)] channel if you have any questions.

</comment><comment><USER11> Being able to query/search/sort on encrypted data sounds really powerful and super relevant for this day and age.

I look forward to playing around with it.

Very cool MacLane!

</comment><comment><USER12> SQL Server has Always Encrypted - [https://msdn.microsoft.com/en- us/library/mt163865.aspx](https://msdn.microsoft.com/en- us/library/mt163865.aspx)

<comment><USER13> That doesn't appear to use homomorphic encryption though

<comment><USER3> Yeah, that is actually deterministic encryption.

Microsoft had a very interesting research paper called CipherBase (we cited it), but I think they didn't end up commercializing that research

</comment></comment></comment><comment><USER14> OK - So why should I use ZeroDb when it seems to be using Crypt-DB for encrypted query processing [which is the USP of ZeroDb] when cryptDb is fully open source, free and verifiable?

<comment><AUTHOR> ZeroDB isn't using CryptDB. And ZeroDB is open source: [https://github.com/zero-db/zerodb](https://github.com/zero-db/zerodb)

<comment><USER14> Thank you for the clarification. How are you computing over encrypted data? From the "acknowledgements" section of the paper, it seems to be the same technique as cryptDB. If so, where is this source code? [I'm assuming this to be in C/C++] in the "crypto" folder as in cryptDb

<comment><USER3> No, we don't use any techniques from CryptDB at all. We thought about doing server-side set intersections using some ideas from CryptDB but we decided to not do that.

In brief, we don't compute on encrypted data, we search for encrypted data using encrypted indexes (which are operated from the client, even though are located remotely).

</comment></comment></comment></comment><comment><USER15> Well...

Guess I need to rename [https://github.com/StabbyCutyou/0db](https://github.com/StabbyCutyou/0db)

</comment><comment><USER16> Also the name of a popular silent disco company in San Francisco:

[https://www.facebook.com/ZEROdBsf/](https://www.facebook.com/ZEROdBsf/)

Clearly not overlapping business though.

<comment><USER3> Yeah, right. I guess, as long as we don't redirect encrypted content of a database to a sound device, we should be all right :-)</comment></comment>


Ask HN: Are avatars so important? - <AUTHOR>

Hi folks,
me and my friend always asked us, how important avatars are on the websites where people participate in discussions. Nowadays all the forums around uses avatars, all the IM clients uses avatars, skype and so on. Yes, even stackoverflow uses them. So, it seems, that everywhere, where people tend to communicate or maybe interact with each other, avatars are used.
(HN doesn't count here, because it goes a really really sparse design paradigm)
However, I also know people which hates them. In some case this is also a good point, since avatars could break the design experience of a website: imagine a pink pig as an avatar on some more serious site, e.g. like stackoverflow. In my case I often just ignore them, however, I found my self putting an avatar representing me on a lot of sites and im-clients.
So, what do you think: How is it important for a website to provide avatars support to their users, if the website is going main stream? What is a benefit of this? Does it increase 'user come-back' ratio? Or is it just a question of: "Everybody uses them, so let us also use them"?
thanks in advance <comment><USER2> "Everybody uses them, so let us also use them"

That's my guess.. the web is very monkey see, monkey do. That said, I don't mind seeing a pink pig on stackoverflow. But I never fill in the avatar images myself</comment>


Ask HN: Junior Android Developer jobs? - <AUTHOR>

Where did all of the junior Android developer jobs go? Why is it so hard to find a job listing that fits this description? Is the market really that saturated with Android developers? It seems that everyone wants years of work experience, regardless of how good you are. <comment><USER2> > Why is it so hard to find a job listing that fits this description?

Are you looking for a job listing, or for a job? Perhaps try to proactively contact companies you think could benefit from hiring you.

> It seems that everyone wants years of work experience, regardless of how > good you are.

You can treat the years of work experience on a job listing however you want. You can think of it as an indication of the experience level they're seeking. If they ask for N years of experience in XYZ tech, and you are better at XYZ than most people who have N years of experience, then (i) apply, (ii) find a way to show how good you are.

For (ii), this should be easy if XYZ=Android, as the software you have worked on (for previous employers or on your own) is something they can run on their own phone. If it can run on a phone then it doesn't need some huge deployment effort or special build environment for them to try.

</comment><comment><USER3> The years of experience isn't a hard & fast rule. For example, as soon as 1 1/4 years after I started working in frontend web development, I was receiving offers for lead & senior engineer positions, some where the job postings say 10+ years of experience. The important things are the quality of experience, and having demonstrable knowledge in your domain - in my case, my contributions to various open source libraries, some major, has unexpectedly skyrocketed my own career.

As for finding a junior position, your best bet is likely to work with a recruiter.

</comment><comment><USER4> I don't have any knowledge of these markets, so I'm just speculating here.

Your query might be a problem. Maybe in a job ad for an entry-level role they're not using "Android" in the text as they presume only "programming", "computer science", "mobile", "devices", or "java" are the skills a junior dev would have.

Also, after putting out a job ad with skills keywords, employers are flooded with crap resumes and every candidate claims to be an expert in said skill.

And maybe companies don't need junior devs devoted solely to Android but need general Java devs.</comment>


Researchers Hijack Storm Worm to Track Profits - <AUTHOR> http://voices.washingtonpost.com/securityfix/2008/11/study_spam_still_profitable_at.html?hpid=sec-tech <comment><USER2> Interesting - so one out of every 12.5 million spam email messages actually works. I wonder what the rate is for the 419 scam

</comment><comment><USER3> If they just had to pay 1 cent per email, the dynamics would be fundamentally changed.

How might one make that happen?

<comment><AUTHOR> Bill Gates (among others) has already tried. <http://www.cnn.com/2004/TECH/internet/03/05/spam.charge.ap/> If he failed, I would say there isn't much hope of getting it done.</comment></comment>


Why Open? Steve Klabnik at API Days - <AUTHOR> http://www.youtube.com/watch?v=oniWVV2k1Pg

<comment><USER2> I love the comparison between web ecosytem and biological ecosystem

</comment><comment><USER3> Interesting post. I think these guys at Webshell are doing a lot of useful things. Wish them all the best for what's next.</comment>


Feedback please for YC rejects - <AUTHOR>

I appreciate that YC get a lot of applicants and that personalised feedback might seem like a daunting task with the number of applications you get but, as you recognise for yourself in your rejection letter by asking us for it when we succeed with our companies, it's very important and highly desired.
Generic feedback is what everyone gives (conferences, job applications, accelerators etc) and is pretty disappointing for applicants as we put so much work up front. If the numbers of applications rumoured are right and you got about 1800, and each form takes on average 2 people-days to complete then as a community we put in 10 years effort just to apply. That's 10 years that for the majority of us could've been put to better use into our companies/ideas instead so it would be nice if we could just get a little back in return to help us be better next time round.
If you really want to differentiate yourself from the pack (and I think you probably do) then do something that no-one else does. Give feedback. For the sake of a few people days extra effort your end it would come back to you a hundred times over in other ways.
I'm sure you have a process for sorting the wheat from the chaff but here's what I would do if I were wearing your shoes:

   1. First pass have two buckets - MAYBEs and NOs. For all the NOs have someone writing down one sentence of why they're a no and include in their rejection letter (via a mail merge off a CSV)
   2. Second pass go through the MAYBEs. For all the NOs this time round answer two questions:
               1. What went well (with their application)?
               2. Even better if (their application had/said/was/included ...)?
 Two/three bullet points for each question would be amazing for us and you'd probably find yourself repeating. You will also probably find it will help generate content for your general feedback document and guides.
Hope my feedback is seen as just that rather than criticism. I love you guys but feedback is an essential part of the learning process and one that I just can't help asking for whenever I see an opportunity! <comment><USER2> If you took 2 person-days to complete the YC application form, you may have agonized over it too much, or hadn't yet done other definitional/pitch work that you'd have had to do eventually anyway. So you can't really charge the whole two days against YC.

If you're really eager for a few bullet points of feedback, send a copy of your application materials to a few approachable people whose opinions you would respect. Or even post it publicly.

<comment><AUTHOR> There was no agonizing at all. It's a long form (and video) needing all founders discussion/input and the questions are tailored and formatted for YCs requirements.

Sending the application to other people or posting publicly isn't going to help understand why YC specifically rejected it either.

Do you think feedback is an unreasonable request?

<comment><USER2> It's reasonable to want more feedback, but given YC's model and applicant volume, it's also reasonable for them to give most applicants a generic decline, as well.

I'd say your reward for the effort was the chance at acceptance, plus the benefits of the exercise – having necessary pitching discussions with your cofounders.

Why focus specifically on why the YC process passed on you? Lots of other people can give feedback that will reasonably approximate YC's – and perhaps more importantly, will let you know what _other_ funders/mentors/customers would think.

The keen desire for a YC-specific-explanation is itself a potential reason they might avoid giving detailed feedback... even a few words might encourage some teams to obsess over YC's estimation (and perhaps keep trying to 'clarify'/'explain' their application) rather than moving forward on other opportunities. Sometimes, a simple 'no' is a favor, too.</comment></comment></comment>


Nothing is certain, except death and taxes .. and chargebacks - <AUTHOR> http://blog.balancedpayments.com/death-taxes-chargebacks-balanced/ <comment><USER2> I'm surprised the part on combatting friendly fraud (4.2) didn't include a part about contacting the customer directly, most likely via phone. If you are in an industry where this occurs more often, you might even want to invest in telephone authentication. While it won't stop the friendly fraud, it will be a deterrent.

Anyways, contacting the customer can usually get things resolved as well, even if they went straight for the chargeback.

Never underestimate human contact. You might be surprised why they went with a chargeback. Some just thing it's the way to get a refund for something that was wrong. Yes, sometimes the person is just being a douche, in which case you can assure the person that you thank them for reporting the case, and that you will be following up by filing a police report. When they realize what information you have available, some are quick to want to work something out, usually that involves calling the bank in a 3 way conference call and canceling the charge back.

It won't always work, but the nature of chargebacks means every little bit helps.

Again, this also depends on the nature of the industry you are in.

<comment><USER3> The best way to combat friendly fraud is extensive logging and 3DS.

<comment><USER2> Yes for logging.

3DS, while effective, is not a bullet proof solution. Requiring 3DS transactions will impact sales, and can impact them enough that it's better to not use it. I always recommended a scoring approach (reach a certain threshold, and we require 3DS).

Even still 3DS only affects the initial transaction. Any recurring payment won't benefit from the 3DS transactions. There are ways to encourage 3DS use (discounted membership fees if one performs a 3DS transaction), but outside of games like that, 3DS only affects the transaction it's made with.

</comment></comment></comment><comment><USER4> Use digital cash- no chargebacks, no middlemen.

The best option today if you don't want chargebacks is adopting Bitcoin for payments (like how Reddit did for Reddit Gold with Coinbase).

[http://blog.coinbase.com/post/40131065845/hosted-payment- pag...](http://blog.coinbase.com/post/40131065845/hosted-payment-pages-email- invoices-and-more)

<comment><USER5> What happens if bitcoins are purchased with a credit card? Who will be responsible for the chargeback in that case? Coinbase?

<comment><USER6> Yes, if a bitcoin exchange accepted credit cards, they'd be on the hook for any chargebacks. Hence why most of the existing exchanges want a bank transfer instead.

In theory, a bitcoin exchange should have a near-perfect defense against chargebacks for faulty or missing products or similar, by showing via the public block chain that they delivered the purchased product as requested. However, there's no defense against chargebacks claiming that the cardholder didn't make the purchase (stolen card number, etc).

<comment><USER5> What if the buyer does a reversal of the bank transfer -- usually possible up to 60 days afterward. (I swear I'm not trolling. I'm new to bitcoin exchanges)

<comment><USER7> ACH transfers are reversible and a few bitcoin exchanges have been bitten by them since most have/do take dwolla, which is a nice service for ACH transfers.

Bank wire transfers, though, are unable to be reversed. Its a pity that most US banks charge for them.

<comment><USER5> Yeah, the US banking system is pretty pitiful in general. We (Balanced) are trying to make it a little bit better by making ACH faster and easier to integrate, but we can't do much about reversals. [shameless plug] Example, checkout our ACH payouts feature: blog.balancedpayments.com/announcing- balanced-payouts/

</comment></comment></comment></comment></comment></comment><comment><USER8> I would actually be interested in knowing how others deal with a certain type of fraud.

We currently have an issue where someone is using stolen credit cards to buy "digital goods".

We in the UK and Scandinavia, so we started out blocking purchases of digital goods from the UK. Fraud goes to zero right away.

The fraudsters moves to using stolen UK credit cards in Denmark, via a large number of Danish IPs, fine... We'll just require that the card is issued in the country where your IP indicates that you're located ( not 100% correct, but close enough ).

At this point fraud has been reduced to zero for a few weeks. The next we really where not expecting. The same pattern of buying starts showing up, seems like fraud and it turns out it is. We now see a stolen Danish credit cards.

At this point we're more or less reduced to having to approve every purchase manually. The only real solution currently is 3DSecure for MasterCard or Verified by VISA. These solutions are very American and not at all what European customers expect to see. Enabling 3DSecure scares of legitimate customers, but it's currently the only solution.

The article looks at high velocity, that does nothing in some cases, if people are out to scam you, they will appear as a new customer for a new IP, with a new card.

CSC are useless, these are stolen all the time.

AVS is supported by almost no one.

Looking a transaction amount compared to the mean doesn't really work when you mostly sell one product at a time.

Recently created accounts are actually a good indication of fraud, but mostly you have false positives.

Blocking high risk countries don't work for digital goods.

Large distance between IP and billing address, doesn't work well in smaller countries, but worth considering. Somewhat difficult to implement though.

High number of card from the same person... That never happens. Our legitimate customer are the only ones that might use different cards. In the case of fraud cards and accounts are often used only once.

It's not that the article is a bad write up, but non of the information will protect you against someone that wants to scam you. Physical products are easier to safe guard, because the bad guy will need to pick it up at some point, digital good is a lot harder to secure.

<comment><USER9> My company (Sift Science) uses machine learning to fight fraud, and we work with customers who sell digital goods. You're right that normal country blacklisting, IP blocking, AVS, CVV, etc. aren't terribly effective.

I think some effective techniques for digital goods are: 1) behavioral signals, such as how long the user spent browsing your site before making a purchase, 2) physical device -- have I seen activity from this particular machine before, even if they're going through a proxy to use a fresh IP? 3) e-mail address -- is it a legitimate domain? an obvious throw-away account?, 4) mismatch between IP and billing info (as you noted).

In general, fraudsters switch tactics with surprising frequency, so I'd highly recommend combining multiple types of data into a machine learning system that will adapt. Otherwise you're going to spend a lot of time tuning rules.

And if you're looking for help, feel free to send me an e-mail: firstname.lastname@example.com. My company deals with fraud all the time. Even if we can't help, I'd be happy to point you to others who can.

<comment><USER10> Brandon's a great guy, very proactive and helpful. We didn't have quite enough volume yet (w/ Gittip) to use his services, but I have a positive opinion of him.

E.g.: <https://github.com/zetaweb/www.gittip.com/pull/387>

<comment><USER9> Thanks Chad! It's a pleasure to work with people like you!

</comment></comment></comment></comment><comment><USER11> Balanced is burying the lede on this, the final table of correlations between payment information signal failures and incidence of fraud is pretty fascinating.

<comment><USER5> A single user attempting purchases with many different credit cards is fraudulent 100% of the time!!

<comment><USER12> This probably assumes that all cards are under different names.

<comment><USER13> The merchant doesn't know what name is on the cards. It's still virtually guaranteed fraud when one person presents more than 2 or 3 cards on your site in a short period.

<comment><USER14> Too bad you can never know what "one person" is.

<comment><USER13> In theory, everything can be evaded. In practice, it won't be. If you run your transactions through something like MaxMind MinFraud with Device ID, you will know it's the same person, even if they clear cookies, switch proxies and re- register on your store between every card. It costs half a penny per transaction; anyone can afford basic risk scoring.

Most of the time that kind of tech isn't even necessary. The types of criminals most online stores deal with are not sophisticated; they're just people that paid $1/number for a list of phished credit cards on a black market forum who are going to enter them one-by-one on a few websites to see which haven't been reported stolen yet.

</comment></comment></comment></comment></comment></comment><comment><USER15> It would be interesting to see how this compares to other merchants and/or other payment processors. Does anyone have another source?

<comment><USER16> (I wrote the post) I am not aware of any. That's actually one of the motivations to do this. Someone needs to start !

</comment></comment><comment><USER17> It's a pity they didn't bother to run it through a spellchecker first.

<comment><AUTHOR> Sorry about that! Just did a proofread for obvious errors and I think I got them all. I'm omw out now, but I'll take another look when I get back.

It's open source, you're welcome to contribute a fix: [https://github.com/balanced/balanced.github.com/blob/master/...](https://github.com/balanced/balanced.github.com/blob/master/_posts/2013-02-21-death- taxes-chargebacks-balanced.md)

<comment><USER17> You're telling me you've crowdsourced your editing for marketing materials?

Tell me you're not serious.

<comment><AUTHOR> It's just part of being an open company, read more here: [http://blog.gittip.com/post/26350459746/the-first-open- compa...](http://blog.gittip.com/post/26350459746/the-first-open-company)

We crowdsource feedback for a lot of things we do @ Balanced. For example, we've openly discussed pricing (<https://github.com/balanced/balanced- api/issues/48>), etc.

Everything's on <https://github.com/balanced/balanced-api>. We're trying a different approach to payments, for once. Openness and transparency.

<comment><USER10> :D

</comment></comment></comment></comment></comment><comment><USER18> I wonder how would Balanced Payments deal with 'item not as described' fraud Something that happened with me here:

<http://news.ycombinator.com/item?id=4867484>

<comment><AUTHOR> Hey there, I'll ping npcomplete to answer this one. He has some thoughts on "item not as described" fraud.

</comment></comment><comment><USER19> That's super weird- I had my first chargeback today, from a customer who didn't even attempt to get in touch and ask for a refund... apparently this is quite common!

<comment><USER8> Not common, standard. Having the customer contact you isn't a frequent occurrence, depending on where you do business. The British do not wish to talk to you, they assume that you're the one trying to defraud them by default. Swedes will pretty much never do charge backs.

<comment><USER20> It always fascinates me when the global nature of software uncovers quirky cultural differences.

</comment></comment></comment><comment><USER21> About the first ones: Unless you are a libertarian transhumanist, I guess.</comment>


Why I switched from Dropbox to Windows Live Mesh - <AUTHOR> http://www.zdnet.com/blog/bott/why-i-switched-from-dropbox-to-windows-live-mesh/3512

<comment><USER2> "Author of more than 25 books on Microsoft Windows and Office, including the recently released Windows 7 Inside Out"(from his bio) suggests a Microsoft solution.

[edit: removed "tldr" tag because this was clearly not a summary but a mere snark.]

<comment><USER3> Ad hominem

<comment><USER2> <blush> It's true, it is.

This is a tangent, but an honest question. The biases and agenda of an argument's author are often essential to evaluating the argument. In the human world, a person would be pretty naive to not "consider the source".

Is it always invalid in written discourse to raise the issue of the author's background, or is there a way to do it that isn't ad hominem?

<comment><USER3> Of course it's valid to consider the source, but you need to back it up, show that it's influencing his opinion unfairly. You may be right, maybe he is just favouring MS products, but then show why his comparison is wrong, or why he is ignoring better alternatives, something at least.

</comment></comment></comment></comment><comment><USER4> In theory, yes. The recent news about Dropbox are not encouraging. The thing is, here is a list of things I need from a syncing service:

* Reliability. No fucking up my files, please.

* Low Maintenance. It should _just work_ , basically.

* Cross Platform. I _need_ to work reliably on Win, Lin, OSX. And at the same time, too.

Anything else is just bonus. If you know a syncing service besides Dropbox that satisfies all three of these, please let me know!

Oh, and bonus points for iOS/Android integration.

<comment><USER5> SpiderOak?

<comment><USER6> I installed SpiderOak on my Mac when the Dropbox security failures came to light a couple weeks ago. SpiderOak is just plain confusing and hard to use. I think I setup a folder to sync but with all of the options and "features" it has, I really don't know what is being synced or not. I'm a software developer by trade, and if I find it this confusing, I would definitely not recommend it to regular people. I'm going to give Live Mesh a shot and see how it works.

</comment></comment></comment><comment><USER7> I used Live Mesh once. Then, out of the blue, after a reboot reboot it started deleting all my files. I got in touch with the Live Mesh team and was told that I should remove files with unsupported extensions (WTF?) from my synced folders and see if my files came back. I think they were eventually able to recover some, but not all, of the stuff that was lost. This was early 2009, so maybe the client has improved since then, but I can't bring myself to trust it.

<comment><USER8> I used Live Mesh from launch up until about a year ago and had similar experiences. Lots of sync errors, lots of pain upgrading when MS decided to shuffle Live Mesh over to the Windows Live team.

On the other hand, Dropbox has been incredibly reliable and easy to use, plus it works on Android and Linux.

</comment></comment><comment><USER9> He makes one very good suggestion:

 _At the very minimum, Dropbox needs to have a thorough security audit from an independent group to ensure that it has the processes in place to back up those promises._

Call it security theatre, a waste of time and resources, or whatever you want, but I would find this very reassuring as a Dropbox user, and it would help me to trust the service again. That, and an independent verification of their claims about the number of users possibly affected by the no-password security bug.

<comment><USER10> I would bet on this being one of the outcomes of the class action lawsuit against them.

</comment></comment><comment><USER6> So I'm trying out LiveMesh on my Mac right now. Installation was easy and so was setting up the sync folder. I copied about 100MB of source code into the folder and it started syncing right away. However, it's just killing my CPU. I have a Core i5 MBP and CPU utilization is 100%-130%. Memory usage is quite low at ~70MB. I don't recall Dropbox killing my CPU like this when doing a simple sync. I wonder if it's doing some sort of encryption?

</comment><comment><USER11> It's interesting that my "emotional" reaction is to favor Dropbox to Windows... I like the brand, the entrepreneur in me is naturally inclined to cheer for the startup versus the corporate giant and I've been telling people about Dropbox making me somewhat more vested in it. That said, security issues _are_ affecting your image Dropbox. Please fix these and I'll keep loving you and have no reason to switch.

</comment><comment><USER12> As an alternative I have never seen Powerfolder (<http://www.powerfolder.com>) mentioned here, and that's a pity. Powerfolder surely is not a nice and shiny start up (German private company, since 2003), but it has worked for us for over 3 or 4 years now. As I understand Powerfolder originally was a client to just sync files and folders across computers but they added "cloud storage" a few years ago, so your devices need not be connected at the same time. The Java client works an Windows, Mac and Linux, it's "functionally" designed, to say it politely (no shiny icons) You can sync backup or share any file or folder on your computer. I've tested nearly all sync and backup services but none of them made me switch, Powerfolder's sync speed and reliability is just superior (they can LAN sync too). And heck if you don't trust "the cloud" you can just buy their server software and set it up on your own hardware.

I still use Dropbox for easy private file sharing, just not professionally.

[Edit: ,just NOT professionally]

</comment><comment><USER13> More technical people like us can just use Tarsnap.

<comment><USER4> The thing is, Tarsnap in and of itself is only a backup, but does not sync. And sync really is what makes Dropbox so great.

</comment></comment><comment><USER14> As suggested recently by 'How To Geek' I have installed Boxcryptor in my dropbox folder.

Boxcryptor was obviously inspired by Dropbox's limitations and as far as I can see solves them perfectly with encryption of every file at the client end.

Dropbox simply stores and syncs these encrypted files.

The linked article says : 'Ultimately, I've come to the conclusion that scale matters in the cloud. Microsoft, Google, and Amazon have the experience and the engineering depth to get operational and security issues right.'

Until a little while ago I wonder if the author would have objected to Sony being added to that list?

<comment><USER15> I don't think of Sony when I think of "the cloud".

However - Microsoft, Google, and Amazon are major players and most people do think of these guys when it comes to cloud storage. Amazon is of course the gold standard and Google is trusted. Microsoft is kinda sort there.

</comment></comment><comment><USER16> It really kind of blows my mind that anyone is willing to use these kinds of cloud-storage tools without their own personal layer of encryption in-between. I've used Dropbox with encfs in the past, and didn't bat an eye when their usage terms changed, because although they were storing my data, they do not have my data. Without something like encfs, you're just literally giving other people your personal files--I don't understand that on any level.

<comment><USER17> Don't you lose the key features of Dropbox as a service: de-dup, sync, share, etc.? In that case, why not just store your stuff directly in S3 and cut out the middle man?

</comment></comment><comment><USER18> I have used both Dropbox and Live Mesh, both work fine for me.

Advantages for Live Mesh: * Free quota is 5 Gb * I can choose the folder I want to sync, so my existing folder structure need not be tweaked

Advantages for Dropbox: * It is more popular so it is easier to set up a shared folder with someone.

</comment><comment><USER19> Why do people split articles into multiple pages like this? Does this increase ad revenue somehow?

<comment><USER20> Yes.

</comment></comment><comment><USER21> Shame Windows Live Mesh doesn't work on Lion. I would have liked to try it.

</comment><comment><USER22> He just spent a whole page talking about security issues and then never mentioned what security he is benefiting from by moving to Windows...

<comment><USER23> Simply put, I am not aware of any heavily publicised security breaches with Live Mesh like there have been Dropbox.

The impression I've had in recent times (something presumably shared by the article's author) is that security is something that Microsoft takes very seriously these days. That has happened out of necessity, originally precipitated by security flaws in the Windows line of OS. But at least they've already learnt that security is a serious issue and tightened things up - something dropbox (and other players from recent hacking scandals) are only just learning.

</comment></comment><comment><USER24> Run it on my XP/Linux desktop.

</comment><comment><USER25> "Why I switched from Dropbox to Windows Live Mesh"

Because I am a Microsoft shill.

<comment><USER26> So you think his other reasons for swapping aren't valid? Or do you think that there is a compelling reason not to use Live Mesh if you want to switch from Dropbox?

<comment><USER25> I got in dropbox very early and they initially gave out 5GB of space, so I have that. His other points are not very interesting.

<comment><USER27> LAN syncing and multiple folder support is fairly compelling. 25 GB web storage sounds pretty good too considering it's free. Although personally I wouldn't use the web storage and would stick to the 5 GB.</comment></comment></comment></comment>


Google Is Making AI That Can Make More AI - <AUTHOR> https://www.yahoo.com/news/google-making-ai-more-ai-175338896.html <comment><USER2> Yes, let's make a black box that we can't understand that makes other black boxes that we can't understand. This is a good idea.</comment>


Networking Guides for Linux Sysadmins - <AUTHOR> https://www.redhat.com/sysadmin/networking-guides <comment><USER2> It's a shame the Amazon/Microsoft marketing machine is slowly killing the art of system administration. Though I guess I probably sound like a greybeard complaining about kids today and their fancy libraries and compilers.

<comment><USER3> It is nice to be able to throw up a nice interactive data viz app in a day on Reddit that can handle millions of POST requests with minimal effort.

<comment><USER4> When it works, it's great. When you have to 'prove' a problem isn't yours, and you send tcpdump files to the developer and partner company's networking team showing how the remote firewall is rejecting the packets, and neither of the recipients knows how a packet syn-ack sequence should work, or what this Wireshark tool is, it's sad.

<comment><USER5> That knowledge is becoming increasingly low level though as we build further abstractions over it. If you're working on a network you probably aught to know how TCP works, or at least know how to use Wireshark and a reference. If you're trying to spin up a simple webapp on AWS, you can pretty much stop thinking about firewalls and networking once you set up security groups/NACLs/policies/etc and confirm you can hit whatever is behind them. With new trends like immutable infrastructure coming into vogue, I'd even argue that system administration altogether is becoming a more niche skillset.

</comment></comment></comment></comment><comment><USER6> RedHat spams the crap out of /r/redhat with these, some of them are of very low quality

<comment><USER7> Indeed, which is what they're doing here: the OP only posts RedHat fabrications like that in here...

<comment><USER8> Yep, one a day for the last month, on average -- and they post the shortened links instead of the "real" URLs.

All but two have ended up _[dead]_

</comment></comment></comment><comment><USER9> >Becoming friends with NetworkManager

Never. The same goes for Systemd too.

<comment><USER10> Why?

[https://www.youtube.com/watch?v=o_AIw9bGogo](https://www.youtube.com/watch?v=o_AIw9bGogo)

</comment></comment><comment><USER11> Lists don't make for such great discussion, because they end up being about the lowest common denominator of the items on the list, which is always something generic. Generic discussions aren't as interesting because there's rarely anything new to say about them. It's better to pick the most interesting item on the list and submit that.

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20denominator%20list&sort=byDate&type=comment)

[https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...](https://hn.algolia.com/?dateRange=all&page=0&prefix=true&query=by%3Adang%20generic%20discussion&sort=byDate&type=comment)

</comment><comment><USER12> Is there any good Linux based networking book/resources which teaches you from first principles ?

<comment><USER13> Mostly linux agnostic, but in case it helps, its was all right for me. The practice exercises at the end of each section offer some hands-on help with some linux tools

[https://s3.amazonaws.com/saylordotorg- resources/wwwresources...](https://s3.amazonaws.com/saylordotorg- resources/wwwresources/site/wp-content/uploads/2012/02/Computer-Networking- Principles-Bonaventure-1-30-31-OTC1.pdf)

as recommended from this course

[https://learn.saylor.org/course/view.php?id=84](https://learn.saylor.org/course/view.php?id=84)

</comment></comment><comment><USER14> A listicle of networking guides. I guess this may be found useful by some, but it appears to me as buzz feed-tier tech writing with no content produced; certainly not what I'd expect from a professional outfit like red hat.</comment>


US government will require all drone purchases to be registered - <AUTHOR> http://www.theverge.com/2015/10/16/9558383/us-government-will-create-drone-registry <comment><USER2> On the one hand, it makes sense to have registries of vehicles passing through public spaces, for the sake of contract law and liability; on the other hand, is there any human activity that the US government will not want to monitor, control, rule, or otherwise restrict?

</comment><comment><USER3> The article states that there were "several recent close calls involving remote-controlled drones and commercial aircraft" but provides no details. Anyone have links to more information?

I've heard of an issue, singular, with a NYC company flying drones too high and into controlled airspace, but wasn't aware that there had been any genuine close encounters between a plane and a drone.

<comment><USER4> [http://www.washingtonpost.com/sf/investigative/2014/06/23/cl...](http://www.washingtonpost.com/sf/investigative/2014/06/23/close- encounters-with-small-drones-on-rise/) \- "In 15 cases over the past two years, drones flew dangerously close to airports or passenger aircraft" \- "A NASA database of confidential complaints filed by pilots and air-traffic controllers has recorded 50 other reports of close calls or improper flight operations involving drones over the past decade." \- "Since November 2009, law enforcement agencies, universities and other registered drone users have reported 23 accidents and 236 unsafe incidents, according to FAA records."

> On Sept. 22, while at an altitude of 2,300 feet over Phoenix, a pilot > reported a near-collision with a black-and-white drone the size of a > basketball, according to records the FAA released with many details > redacted. The pilot reported that the drone was 200 feet ahead and closing > in. The pilot swerved left and the two aircraft missed each other by 50 > feet.

> On March 25, 2012, a pilot was flying 11 miles northwest of Houston at 2,000 > feet when he saw what he described as a drone just 100 feet below his plane. > The mysterious aircraft disappeared in a blur before the pilot could get a > better look. He notified the control tower, but it could not find the drone > on radar.

<comment><USER3> Interesting, thank you very much for that link.

I gotta say, though... Back in the day, I was really interested in skeptic culture, which at that point was focused on debunking UFO sightings. Often, airline pilots were the ones who reported UFOs. And the pilots' drone reports in that newspaper article really feel familiar to the old UFO reports in their vagueness and impossibility to confirm, without useful radar information or photographs. This one in particular set off screaming alarm bells for me:

"In the first incident on May 29, the pilot of a commercial airliner descending toward LaGuardia Airport saw what appeared to be a black drone with a 10-to-15-foot wingspan about 5,500 feet above Lower Manhattan, according to a previously undisclosed report filed with the Federal Aviation Administration."

10-to-15 foot wingspan? 5,500 feet up? Really? And nobody could track down this enormous mystery aircraft operating high above one of the most tightly controlled airspaces in the world and almost colliding with an airliner?

Of course, none of this says that a drone and an airplane couldn't get into a fatal collision at some point -- it's a serious danger we should consider ways to avoid -- but there's just something about these reports that feels terribly off.

<comment><USER4> I can see where you are coming from. Plus, these observations are of "unidentified" flying objects, making them technically UFOs.

When will airlines have the equivalent of dashcams?</comment></comment></comment></comment>


Blinked.ca, A bitmessage web client with in browser encryption - <AUTHOR> https://bitmessage.org/forum/index.php?topic=4059.0

<comment><USER2> I made that! It was a shock to see it on here, thanks for posting.

<comment><AUTHOR> Of course, this is a great way to show what's possible with bitmessage! And it seems to work on my phone (with a couple minor ui glitches in mobile firefox).

On the desktop, Tor + Blinked is a very easy solution to basic anonymous communication.</comment></comment>


Ack is designed as a replacement for 99% of the uses of grep - <AUTHOR> http://linux.die.net/man/1/ack <comment><USER2> The Silver Searcher, or ag: [https://github.com/ggreer/the_silver_searcher](https://github.com/ggreer/the_silver_searcher)

Designed as a replacement for all of the use cases of Ack. Is even faster.

<comment><USER3> I'll engage in some expectation management: ag doesn't cover _all_ uses of Ack. I made ag because I loved Ack, but it wasn't fast enough for my needs. For the features of Ack that I use, ag is interchangeable. For features I don't use... not so much. Also, since the release of Ack 2.0, some behaviors and command line options have differed slightly.

If you want to know why ag is so fast, there's a summary in the GitHub README.[1] I've also written a few blog posts on the subject.[2]

1\. [https://github.com/ggreer/the_silver_searcher#how-is-it- so-f...](https://github.com/ggreer/the_silver_searcher#how-is-it-so-fast)

2\. [http://geoff.greer.fm/ag/#related- posts](http://geoff.greer.fm/ag/#related-posts)

<comment><USER4> I'll take this opportunity to thank you for this productivity enhancing great piece of software!

</comment></comment></comment><comment><USER5> While "would you rather type" examples are OK, I wish more people would learn how to use their shell's short-cuts and trivial scripting.

Whenever you find yourself repeatedly running long and awkward commands that you wish were shorter, you ought to think about converting them into an alias or a script.

</comment><comment><USER6> "Ack is designed as a replacement for 99% of the uses of grep."

Alright, but where is the advantage of it?

<comment><USER7> The name is one letter shorter. Here are some other advantages listed: [http://beyondgrep.com/why-ack/](http://beyondgrep.com/why-ack/)

<comment><USER6> Thanks, though I hope that statement came with a pinch of sarcasm ;)

The list reads more like a list of features though than "advantages over grep".

<comment><USER8> About the command name length, probably a running joke ?

This is from the feature list of the Silver Searcher, listed above:

"The command name is 33% shorter than ack, and all keys are on the home row!"</comment></comment></comment></comment>


Hitwise Data Pegs Panda Impact On Demand Media Sites At 40% - <AUTHOR> http://searchengineland.com/hitwise-data-pegs-panda-impact-on-demand-media-sites-at-40-74640

<comment><USER2> As long as they can actually address this by improving content then the update is a success because it pushes demand media to generate better content and remove the low quality stuff they have.</comment>


The Science of Addictive Junk Food (2013) - <AUTHOR> http://www.nytimes.com/2013/02/24/magazine/the-extraordinary-science-of-junk-food.html?partner=rss&pagewanted=all&_r=0

<comment><USER2> What's frightening is the conservative/libertarian view:

"Government has no business telling me what to eat."

Which is just nonsense, and not only because the decision on what to eat is driven by genetics that did not evolve in the presence of modern processed foods. Government isn't telling you what to eat if it keeps companies from selling the worst junk food. If you want to make junk food yourself, you're welcome to (and it will probably be a lot healthier than what you buy, too, and taste better, subjectively, because you made it), but the problem with junk food is the economy of scale. Companies can cheaply make stuff that tastes great without being healthy, so why would you bother with more expensive or healthier food, or anything you have to spend time making yourself? It's about shifting incentives, not banning anything. They counter that freedom applies to corporations too, and you're banning corporations from doing things, which leads to the "do corporations deserve the same freedoms?" argument.

The conservatives' position goes beyond unwise and into the realm of hypocrisy, because conservatives have no problem when government wants to tell me I can't grow salvia divinorum (yes, in my state it's banned), or poppies, or marijuana plants, but forbid a _company_ from distributing disgustingly unhealthy lunchables or soda and they lose their minds.

(I should add that another part of the problem is that both conservatives and libertarians pretend the costs of eating badly only affect the people making the choice, which isn't true. Particularly with the changes to healthcare, everyone buying their own insurance during open enrollment, and maybe everyone (I don't know, are there any plans where premiums are segmented by behavior other than tobacco use?), is in a gigantic pool where no separation by diet is allowed, even if it were practical which it's not. And if everything were privatized and pools segmented by diet, there would still be costs to others, because people in those pools have more health problems that negatively affect not only their own, but their relatives' and friends', happiness and productivity. Which works if their friends in turn exert influence back to eat healthier, but that)

<comment><USER3> The government is not capable of 'shifting incentives' through the use of force. Attempts made for a small group to dictate what others want and what's best for them will, and have, ended in failure.

<comment><USER4> Governments have very effectively reduced tobacco use through advertising, restrictions on where smoking is legal (backed by force), and taxation (backed by force). Making something expensive and inconvient can make it unpopular while avoiding potentially counterproductive bans. The same principles could be applied to unnaturally delicious processed foods.

<comment> norea-armozel I would hazard a guess that the decline in celebrity endorsement of tobacco products had a larger influence on the total decline of smoking at least in the adult population. Mind you, this wasn't uniform as the rate of smoking has not declined at the same rate for poorer populations despite the absence of both advertisements or celebrity endorsements.

</comment></comment></comment></comment><comment><USER5> tl;dr anyone?

<comment><USER6> Free market capitalism works really efficiently, except when externalities are present, at which point it efficiently exploits them to everyone's detriment. Your health is an externality to the corporations that make your food.

<comment><USER7> Brilliant. But completely incomprehensible to almost everyone.

<comment><USER8> "Sometimes it _IS_ more profitable to poison your customers, even in the long- run."

</comment></comment></comment></comment><comment><USER9> How many millions of life-years have been lost due to the junk-food industry? To me they are murderers with expense accounts.

<comment><USER10> It is not that simple thought. The simple $1 McDonalds burger has one of the best value/price ratio in the history of the human nutrition.

"The double cheeseburger provides 390 calories, 23 grams of protein – half a daily serving – seven per cent of daily fibre, 19 grams of fat and 20 per cent of daily calcium, all for between $1 and $2, or 65p and £1.30, The Times reported."

[http://www.telegraph.co.uk/foodanddrink/10210327/McDouble- is...](http://www.telegraph.co.uk/foodanddrink/10210327/McDouble-is-cheapest- and-most-nutritious-food-in-human-history.html)

I think this topic is similar to 'how many life-years have been lost due the pollution caused by industrial revolution' \-- many, of course. And many have been saved as well. Here we can surely say that many life years have been saved by the mass/junk food industry as it has provided the masses reliable, usually not infected/spoiled food.

<comment><USER11> I usually bring up a similar point when people criticize modern agriculture and food production.

Right now someone can buy a pound of ground beef for around $3-4. After tax that represents around 30 minutes of labor for a minimum waged employee.

It is absolutely remarkable that our lowest skilled, lowest productivity workers, can afford a pound of beef for 1/2 an hour of work.

Obviously there are clear environmental issues here, and we are not taking into account all the externalities associated with meat production. But that figure above, and what it represents in terms of human nutrition, should be breathtaking.

<comment><USER12> That's only because we're exploiting other living species, which are much less able to defend themselves than we are.

A society should be judged by how it treats its weakest members, and if you ask me, that's the battery hens and mass-farmed pigs in their crowded "flats". The atrociously low price of meat is actually not at all a mark of civilisation, but rather of barbarism towards animals. Animals which, might i add, consistently turn out to be more intelligent that we originally thought.

Everybody should read _Eating Animals_ by J.S. Foer, it's surprisingly level- headed and non-ranty, but even so, it's hard to ignore what goes on in the food industry.</comment></comment></comment></comment>


Help hack the Zeo Sleep Coach data - <AUTHOR>

I got the Zeo device after reading about it in an article linked off of Hacker News. I assume I'm not the only one, so I turn to other HN readers that got Zeos to help me hack the data stored on the device's SD card.<p>If you want to help, please submit your data (code / thoughts are always welcome, of course) at http://github.com/costan/zeo_sleep_decoder/ <comment><AUTHOR> This is my first HN post. If it's off topic, I apologize. Please let me know, and I'll learn my lesson.

<comment><USER2> It's better to title self-sourced posts as 'Ask HN: help me hack...'. Also, perhaps you could explain briefly what your objectives are. Lots of luck!

</comment></comment><comment><USER3> You can get cheaper handheld ECGs from medchoice/medchina. think 300$ or thereabouts. Surf the recent American Telemedicine Associations conference at the Rio in las vegas for more options.

<comment><USER3> Here's a fun one: [http://www.alibaba.com/product- gs/219038772/Handheld_ECG_mac...](http://www.alibaba.com/product- gs/219038772/Handheld_ECG_machine.html)

</comment></comment><comment><USER4> BTW, if u hack it, you can use it for non-sleep related data too, as a general EEG to wear while programming etc.

</comment><comment><USER5> Shouldn't you also post what the data looks like when viewed or something?</comment>


A Television of Her Own: On Emily Nussbaum - <AUTHOR> https://lareviewofbooks.org/article/television-emily-nussbaum/ <comment><USER2> I am happy to see a piece about Emily Nussbaum here. While it may not be typical "Hacker News", Emily's writing and perspective are top-notch. Her analysis of The Sopranos following its finale over a decade ago is included in her new book and was a spot-on take on the controversial ending: [http://nymag.com/news/features/33517/](http://nymag.com/news/features/33517/)</comment>