Python consumes a lot of memory – how to reduce the size of objects? - <AUTHOR> https://habr.com/en/post/458518/ <comment><USER3> The article doesn't mention a hidden gem in Python's standard library - <USER2> arrays!

Those are in package `array` and they are a very barebones version of numpy's ndarray - with just one (explicit) dimension and no overloaded operators. But if you just want to keep a bunch of numbers in a contiguous array, they can save you tons of memory.

(I know the purpose of the article is to describe a more complex data structure, but arrays can still get you very far.)

<comment><USER4> That's how you deal with that problem with Java as well: arrays of primitives. If you need to operate on million of points, you can either use Point[] which will incur something like 16 MB additional memory or use two int[] arrays which won't incur any extra overhead short of few bytes. Your code won't be pretty, but it'll be fast and you can always hide this weirdness behind pretty API.

<comment><USER5> Also known as the "Struct of Arrays" approach.

<comment><USER6> "Struct of Arrays", to be contrasted with "Array of Structs", because in Java it's actually "Array of pointers to Struct". In languages where "Array of Structs" is actually possible, the decision of which to use is less clear-cut and depends on how big the Struct is, what the access patterns are like, and whether you're trying to perform SIMD operations on multiple Structs at once.

<comment><USER4> That's not only about pointers. Every Java object incurs an overhead of around 16 bytes (architecture and JVM dependent, of course, but it's there).

</comment></comment></comment></comment></comment><comment><USER7> It's not always an option, but simply using PyPy can massively reduce memory usage without needing to change your code at all.

The PyPy site links to this blog post (from 10 years ago) with some info: [https://morepypy.blogspot.com/2009/10/gc- improvements.html](https://morepypy.blogspot.com/2009/10/gc-improvements.html)

And a quick search found this relatively recent post that does some measurement of it: [https://dev.nextthought.com/blog/2018/08/cpython-vs-pypy- mem...](https://dev.nextthought.com/blog/2018/08/cpython-vs-pypy-memory- usage.html)

<comment><USER8> In the article consider aproach with the `dataobject` from `recordclass` library as a base object in class definition. This seem can produce less memory than PyPy.

</comment></comment><comment><USER9> Any type that you have "a lot" of, is a bad candidate for a class/object. Your OO program should typically never instantiate thousands of heap allocated objects at once. A triangle mesh is a good object candidate. One triangle or vertex is not. A device, a render system API handle and an image is a good candidate, one pixel is not, and so on.

So don't use a lot of objects. In C# you use a struct and arrays of them to avoid creating an object on the heap per array entry. In java you have to resort to SoA, in Python it's the same. Just because you have objects doesn't mean everything should be an object. Even languages that follow an "everything is an object"-design usually have an escape hatch such as plain value arrays.

<comment><USER10> It's not about using objects or not, it's how you compose them. A C# array of integers is an object. Each integer in that array is also an object (though granted, it's a special case). Numpy arrays and javascript typed arrays as well.

<comment><USER9> Primitives and value types are not objects in C#. The array is one object, that's it.

<comment><USER10> It has methods and fields, it implements interfaces, and descends from the 'Object' type.

The fact that it's unboxed is an implementation detail.

[https://docs.microsoft.com/en- us/dotnet/api/system.int32?vie...](https://docs.microsoft.com/en- us/dotnet/api/system.int32?view=netframework-4.8)

<comment><USER9> What's relevant for performance is whether an array of 1000 "things" require 1 or 1001 allocated objects, whether accessing thing N in the array requires dereferencing one or two pointers, and whether the item has a storage of 32 bits without overhead for headers. Which ones to call "objects" is semantics.

For efficient access, the array must be a consecutive array of primitives. This is the case in both C# and java for integers. In C# it's also the case for an array of Vector2 with 2 primitives each, which isn't the case in java.

My point is this: avoid heap allocating many things in collections. They must be raw (primitive, consecutive) data without per instance overhead and of course without heap alloc/GC cost.

</comment></comment></comment></comment></comment><comment><USER11> While everyone haggles about the internals I have an interesting anecdote about the costs of tools libraries.

A small service that landed in my lap needed to read(only) a data source that was roughly 10k lines of yaml. No way that was going to be in any way efficient so I asked for suggestions. All the work-a-day devs(I am not) instantly said the same thing without a single real thought about it: Make it a database duh!

Long story slightly less long, Loading up the libraries to interface with a database ate between 2 and 3 times the memory(depending on the DB and lib) that simply loading the entire 10k line yml ate and offered slower performance and required more code.

SQLite was pretty darn close but in the interest of saving developers from themselves vis a vis parameterized queries, or the need to queries all together for that matter, increased the required code for zero benefit.

The service still hums along with a 10k line yaml in memory. "Worse is better" indeed.

<comment><USER12> You could of improved it with a protobuf or a json file then, but with the size of your data set, it shouldn't really matter what your using.

It can be hard to beat an in memory data structure when your data set is small enough, true.

</comment></comment><comment><USER13> In the pure Python cases the 8 bytes per attribute are just pointers. The x, y and z are themselves full-blown objects with all the extra memory overhead that comes with it and this is not counted in the article. For example, an int object uses 28 bytes, so three of them already use up more than each of the described container objects.

The Cython and Numpy cases directly store the actual data and this has the larger effect to reduce memory.

<comment><USER14> On the other hand, the first 256 integers are singletons in python, so they aren't duplicated.

<comment><USER15> Slightly pedantic correction: This is a performance optimization in CPython. I wouldn't be surprised if other implementations have something similar, but to my best knowledge, this behaviour isn't part of the standard.

<comment><USER16> > ... but to my best knowledge, this behaviour isn't part of the standard.

To the best of my knowledge, Python isn't one such language with a standard, is it?

<comment><USER17> The standard is documented in PEPs[1]. CPython is the flagship implementation, and many PEPs discuss it directly, but if you're looking to write another Python, that's where you'd go.

[https://www.python.org/dev/peps/](https://www.python.org/dev/peps/)

</comment></comment></comment></comment></comment><comment><USER18> Many workloads do not require more than 4GB addressable RAM per process. Linux offered a 32-bit user space with 64-bit instruction set:

[https://en.m.wikipedia.org/wiki/X32_ABI](https://en.m.wikipedia.org/wiki/X32_ABI)

As effectively many Python workloads are usually object oriented business logic and objects are mostly pointers, setting up x32 user space "halved" the memory usage. It also made execution performance faster because of better CPU cache utilisation.

Sadly, x32 was "very custom" and very hard to support. Last I heard x32 is being phased out from Linux kernel.

<comment><USER19> Another option is to use the standard 64 bit ABI, but store object pointers compressed into 32 bits when on the heap. This lets you address perhaps 32 GB in a 32 bit value.

<comment><USER20> Could you elaborate more on how the 'compress' part works. Quite curious. I can imagine working with a base pointer and 32 bit offsets.

<comment><USER18> Also curious about this.

Does 64-bit instruction set provide some segments or functionality form this? How about "native" pointers coming from glib and such?

If there has to be base + offset translation on every pointer access it is way too slow.

I would also assume JavaScript VMs in browsers would be already utilising this, as web page workloads are not gigabytes (hopefully).

<comment><USER19> > If there has to be base + offset translation on every pointer access it is > way too slow.

It does do this, but it's not too slow - the overhead of the translation is lower than the benefit of reduced memory transfer, increased cache space, etc. Obviously - otherwise people wouldn't be doing it.

</comment></comment></comment></comment></comment><comment><USER21> I am a C/C++ programmer mostly. Is there good documentation for python development when I care about performance and memory footprint out there as a book or something that anyone can recommend? It was fun reading this but I like to know more.

<comment><USER22> I'm a Python "snob", going on 24 years and If I ever "I care about performance and memory footprint" I'd not use Python.

Python is good enough 90%. You get faster 2x 10x, etc code by picking better algorythms or solutions. In Python, you should not be caring about 10% or 30% speed improvement. It's not worth it, It's not Python's strength.

When you need faster you go to C based libs (the python libs that need to go fast are already in C), [https://en.wikipedia.org/wiki/Cython](https://en.wikipedia.org/wiki/Cython) [https://en.wikipedia.org/wiki/Numba](https://en.wikipedia.org/wiki/Numba) etc.

<comment><USER23> Or go for [1]pypy or [2]graalpython. I know that equivalent code in graal java ee produces equivalent assembly/performance to C+current gcc, which really shows that yes changing to a JIT can pay off. The python in graalvm is early stages but it shows that the old lesson of just write the hot spot in C is no longer true. Chris Seaton shows that for Ruby/C the language Ruby interpreter/C switch is expensive. I think the same is true for Python interpreter/C switches. Something that GraalVM+Python|Ruby can optimize out.

[1] [https://www.pypy.org](https://www.pypy.org) [2] [https://www.graalvm.org/docs/reference- manual/languages/pyth...](https://www.graalvm.org/docs/reference- manual/languages/python/)

<comment><USER24> LuaJIT has great performance through the C FFI. It's more that traditional JITs can't optimize across FFI boundaries. Hopefully Chris will correct me if I'm full of shit.

There are other approaches which can work too, like compiling C to LLVM IR using Clang so a function can be inlined by an LLVM based JIT at runtime.

</comment></comment></comment></comment><comment><USER25> >A significant reduction in the size of a class instance in RAM is achieved by eliminating __dict__ and__weakref__. This is possible with the help of a "trick" with __slots__:

What is the downside to the __slots__: trick? In what cases do you need the dict and weakref?

<comment><USER26> If you are writing code that adds or removes properties at runtime. I.e., when you want to write shoddy code.

<comment><USER27> I have written code like that in the past, and would consider doing so again in the future. When, it's appropriate, it can be hugely beneficial.

When I last did it, I was wrapping a C++ API that I needed to compare against another dataset for a merge. The C++ API (which, admittedly, I also wrote), didn't have equality or hash operators defined on the Python objects. So, I monkey-patched them in for the keys I needed. It was actually the most elegant solution I could come up with as I could then naturally use the objects in sets and easily get differences to update the appropriate dataset.

As an aside, when I wrote the python wrapper around my C++ API, I purposely didn't define equals and hash operators for the objects, despite having full information to do so on the natural keys, because I wanted the flexibility to do the monkey-patching and override how the objects were compared depending upon circumstance.

<comment><USER26> Pretty sure you can still monkeypatch the class instead of each instance, but I'm too jet lagged to think it through at the moment.

</comment></comment></comment></comment><comment><USER28> TXR Lisp, 64 bit:

      1> (defstruct blank ())
      #<struct-type blank>
      2> (pprof (new blank))
      malloc bytes:            16
      gc heap bytes:           32
      total:                   48
      milliseconds:             0
      #S(blank)

32 bit:

      2> (pprof (new blank))
      malloc bytes:             8
      gc heap bytes:           16
      total:                   24
      milliseconds:             0
      #S(blank)

The structure instance has a pointer to its type, followed by a numeric ID (which is also in the type, but is forwarded to the instance for faster access). The ID is combined with a slot symbol to perform a cache lookup to get the offset of a slot. The numeric ID is a fixnum, which leaves a few spare bits for a couple of flags:

      struct struct_inst {
        struct struct_type *type;
        cnum id : sizeof (cnum) * CHAR_BIT - TAG_SHIFT;
        unsigned lazy : 1;
        unsigned dirty : 1;
        val slot[1];
      };

If someone wanted to shrink this, they could patch the code so that the dirty flag support, and lazy instantiation of structs is made optional (as in compiled out), and so is the forwarding of the inst->type->id to inst->id. This struct type is not known outside of struct.c, which is only some 1700 lines long. If you take out the id member from struct_inst, the C compiler will find all the places that have to be fixed up; literally a 15 minute job.

I can also think of a more substantial refactoring that would eliminate the type pointer also. There is no room in the heap object handle to store it directly: heap handles have four words in them; the COBJ ones used for structs have the type field, a class symbol, a pointer to an operations structure, and a pointer to some associated object (in this case struct_inst). All structures share the same operations structure. However, if we dynamically allocate that operations structure for each struct _type_ , we could stick the type pointer in there, taking it out of the instance. Thus an instance size could literally just be sizeof(pointer) * no-of-instance-slots.

</comment><comment><USER29> I am paying US$5 a month for python's memory hogging.

Running Mailman it will not run in 1GB of ram (the US$5 VPS) so I had to give it 2GB. 2GB? For a mail list server?

Had the same problem running motioneye. It will not run on a Raspberry PI Zero. It is advertised to run on a Zero (Apparently I could squeeze it on by doing some Python magic....)

What total crap is Python! What waste, what hubris, what technical failure!

I suspect the problem is actually Django - what hubris! NIH! lighttpd is running sweetly, with some rust templating in a acceptable fraction of the memory.

<comment><USER30> But moving up your stack of desire, what alternate to mailman would you live with? Five a month is $60 a year. Assuming you value your own time at professional wages, this job is worth an hour of your time at most before the opportunity cost of complaint is cheaper than replacement.

<comment><USER31> _opportunity cost of complaint_

No, what he actually bought for his $60/year was license to complain about mailman & python :)

<comment><USER30> The python complaint licence fee was the best deal I ever made.

</comment></comment></comment></comment><comment><USER32> Seems to be an error in editing:

> ...which received a rating of [stackoverflow] > ([https://stackoverflow.com/questions/29290359/existence-of- > mu...](https://stackoverflow.com/questions/29290359/existence-of-mutable- > named-tuple-in) -python / 29419745). In additioobjects liken, it can be used > to reduce the size of objects in RAM...

</comment><comment><USER30> Is this trading space for speed or is there actually a speedup as well in some cases? With interpreters it's possible you be small and fast if you become perhaps obscure or more rigid in your structured definitions

</comment><comment><USER33> This is a pretty neat and useful comparison of how much memory different structures use in Python to achieve roughly the same goals.

</comment><comment><USER34> I always hate how these things are phrased. The idea of "using a lot of memory" doesn't exist in a vacuum. "A lot" relative to what? Do you require run-time dynamic typing features? Do you want to leverage the Python data model? If yes, then this is the memory cost of your desires.

Advice in many of the comments about other languages sounds so tone deaf. The question is not about using less memory. It's about getting _exactly_ the feature set of Python while using less memory.

</comment><comment><USER35> I feel like the last two are cheating a bit by explicitly using 32 bit integers where the other examples seemed to use 64 bit.

<comment><USER36> No, the fields that take up 8 bytes are pointers to PyObject. (I guess this article assumes a 64-bit memory model.)

</comment></comment><comment><USER37> OP, if you're reading this, your article has damaged syntax at the word `additioobjects`.

<comment><USER8> The author was fixed this.

</comment></comment><comment><USER38> this functions u will also see in the case of JAVA s well.this problem arises due to large no. of objects are active in RAM durig the execution of a progrm especially if there ae restrictions on the total amount of availablle memory</comment>


Need a study break to refresh? Maybe not, say Stanford researchers - <AUTHOR> http://news.stanford.edu/news/2010/october/willpower-resource-study-101410.html <comment><USER2> The article sort of pits this study and theory of ego depletion as mutually incompatible, but I think that's probably an exaggeration, either by the author of the article or the authors of the paper. That beliefs can influence ego depletion is not a new finding, nor is it inconsistent with there being a physiological basis.

It doesn't even really sound like they are addressing the theory itself but some particular implication. For instance, the belief that you need to take frequent breaks while studying does not follow directly from the belief that self-regulation is a limited resource.

Personally, as a "believer" of the latter I find the former contradictory: it is exactly what I would _not_ do. In my view, since self-regulation is limited, taking frequent breaks is foolish because means more occasions at which you're forced to self-regulate.

I don't think I'm alone in finding that it requires significantly less willpower to continue a task already in progress than it does to re-initiate an interrupted task. Starting (and re-starting) is always the hard part. I only break when it is patently obvious that the task has already been interrupted. Purposefully taking additional breaks seems like an absurdly premature optimization extremely likely to backfire. And yet, from the sound of it, that's what they advised one group to do. Does this mean I secretly believe in unlimited willpower?

<comment><USER3> "It doesn't even really sound like they are addressing the theory itself but some particular implication."

Agreed. This is just another priming study-- NOT a willpower study. Here's another:

"But the priming effect does not just affect our behavior. It can also, incredibly, have significant effects on our actual performance. In another study cited by Gladwell, two groups of students were asked to answer forty-two questions from the game Trivial Pursuit. But first, one group was first asked to sit and think about professors, the other to think about soccer hooligans. And the difference was dramatic: an average of 55.6% of the questions were answered correctly by the first group, while only 42.6% were answered correctly by the second. How can these results be explained? Can priming somehow temporarily infuse the brain with an expanded store of facts? Obviously not. Instead, what seems to be happening is that encouraging people to "think smart" briefly increases mental qualities like their ability to focus, their sense of recall, and their ability to quickly and correctly integrate diverse pieces of information. In other words, priming does not make us more intelligent, but it does briefly make us better at using the intelligence we already have." [1]

The scarier one is about race:

"The psychologists Claude Steele and Joshua Aronson created an even more extreme version of this test, using black college students and twenty questions taken from the Graduate Record Examination, the standardized test used for entry into graduate school. When the students were asked to identify their race on a pretest questionnaire, that simple act was sufficient to prime them with all the negative stereotypes associated with African Americans and academic achievement - and the number of items they got right was cut in half..."

1: grabbed from <http://www.daylightatheism.org/2007/01/priming-the-mind.html>

</comment></comment><comment><USER4> It looks to me as if what the study shows is not _that willpower is not, after all, a limited resource_ but _thinking of your willpower as a limited resource makes you have less of it_. That's quite interesting (though not, I'd have thought, a big surprise) but not nearly as interesting as it would be if they'd actually shown that it's _only_ thinking of willpower as limited that limits it.

<comment><USER5> Practice makes perfect.

</comment></comment><comment><USER6> Funnily enough, I actually take a nap sometimes when I can't figure out the solution to a problem or I'm going around in circles.

Perhaps it's not realy sleep, but more often than not I wake up with the solution, or at least more ready to tackle the solution.

I've also found almost ubiquitously that whatever had me completely confounded the night before is a hell of a lot easier to work out in the morning.

Perhaps it's just me but I find that sleep for me is a natural reaction to mental stress, and I always feel better afterwards.

<comment><USER7> It think these are two different things.

Our brains can go on working on the problem even if we take a rest. If we rest with a the problem in our head, we are actually still focused. Probably, relaxing and letting the brain do its job without forcing it into one direction, often results in a good solution. Same with me, btw.

The study, as I have understood it, is more about giving up with a problem. You give up faster, if you think your resources are limited. It is more the contrast between either you relax in order to solve a problem, or have some beers with your friends talking about soccer and thus escape.

</comment></comment><comment><USER8> > _"But if you think of willpower as something that is not easily depleted, > you can go on and on."_

That's the problem. Going on and on and on. Usually when it comes to any complex task, going on and on and on is a BAD thing. Forcing yourself to take breaks at a set interval makes you realize and optimize what it is you have actually been spending time on. And assess the next direction to take.

Working without taking breaks is like jamming down the accelerator without steering! You need to do _both_ if you hope to do anything more complex than travelling in a straight line.

"Willpower" is a bit fuzzy and may not be a limited resource, but attentional capacity and working memory most certainly are.

<comment><USER9> Yeah, the article has nothing to say about the effectiveness of the work being done, just that you are "working". I can bang my head against a problem all day long, but I'll find the answer in the shower.

Work smarter, not harder, right?

</comment></comment><comment><USER10> So, if I just BELIEVED that I could keep studying last night for my algorithms exam today, I could have stayed up all night and had a good study session? I wish I had believed that...

<comment><USER11> Not sleeping is counter productive to the goal of learning and retaining information.

</comment></comment><comment><USER12> Study breaks are not about replenishing willpower. They are about giving the brain some quiet time to organize the newly acquired information. Apparently, mental activity interferes with this organization process. I remember covering this 20 years ago in my psych classes. Researchers actually had one experimental group that reviewed an outline during their "break," another that took a nap, and another that was sedated. The sedated group actually retained the most information!

Now this is such common knowledge, it's hard to find the original research with a Google search. This makes me suspicious. Anyone have a reference?

</comment><comment><USER13> What can I say, sometimes by 6AM my edge is considerably dulled, and a bit of a break can help fight off total exhaustion.

There's also writer's block-ish occasions where you keep staring at one problem and it's like beating your head on a brick wall, but if you step away and have a cup of tea and come back, that tough problem unravels itself in front of you.

I've also used the "I need a break" when I know that I'm simply out of juice for the day, and not going to make any more progress (particularly when I have something else heavy on my mind). It's not a break, I don't come back to it for at least a day, but it lets me walk away without feeling like a failure.

<comment><USER14> I've spent hours looking at simple statics problems that I know I should be able to solve in fifteen minutes, but just can't for whatever reason. It's like I've gotten on the wrong mental track early on and can only move further along that track or backwards. I'll give up for the moment to go eat or see some friends or just make contact with the outside world for an hour or so and during that time, the answer presents itself without much effort. I can then come back and redo the problem within 20 minutes.

I wonder, though, if these epiphanies could have happened without the hour long struggle preceding it. Leaving a problem, it feels as if my mind is untangling itself and organizing this mess of thoughts.

What I would really like to see some research on is the mental barrier between myself and starting the work in the first place. I'm looking at it right now... :|

</comment></comment><comment><USER15> <<"If you think of willpower as something that's biologically limited, you're more likely to be tired when you perform a difficult task," said Veronika Job, the paper's lead author. "But if you think of willpower as something that is not easily depleted, you can go on and on.">>

This really bothered me. Maybe if you think that willpower is not easy depleted, it is because you have more of it than the people who think it is.

This being said, I personnaly believe that if willpower is limited, it can at least be trained to seem like it isn't.

<comment><USER16> They covered that objection in the paper, which is available free on the website of one of the authors: [http://www.stanford.edu/~gwalton/home/Publications_files/Job...](http://www.stanford.edu/~gwalton/home/Publications_files/Job,%20Dweck,%20%26%20Walton,%202010.pdf)

In study #2, they manipulated people's beliefs about willpower by administering a "push poll", and then tested willpower depletion.

<comment><USER17> _In study #2, they manipulated people's beliefs about willpower by administering a "push poll", and then tested willpower depletion._

How did they determine how well the push poll influenced that belief? How did they ensure it didn't work better on those who already had less willpower?

<comment><USER16> Did you not want to read the paper? It's really short and clearly written.

They just randomly assign people to group A or B. Group A is told about the actual theory that willpower is a finite resource ("Working on a strenuous mental task can make you feel tired such that you need a break before accomplishing a new task"). Group B is told a competing theory that willpower _compounds_ ("Sometimes, working on a strenuous mental task can make you feel energized for further challenging activities"). They administered surveys to confirm that they had successfully influence each groups opinions.

Then, the experimenters tested the degree to which using willpower became harder or easier throughout a task. On average, group A exhibited willpower fatigue but group B did not.

</comment></comment></comment></comment><comment><USER18> I'd like to see more research into this, but perhaps they can try it out on young children who haven't yet formed solid beliefs about the nature of willpower (or even contemplated it for that matter). You can even teach groups of them one theory or another as separate variables to agnosticism. I still believe willpower is a limited resource but I'd bet people's beliefs about it can lead to underestimating the limit. (Self-fulfilling prophecies play in too.)

</comment><comment><USER19> I still think that frequent breaks are important as they help change thinking patterns.

<comment><USER20> Is that a good example of what the article states though? _"The belief in it is what influences your behavior."_

<comment><USER19> No, I don't think so. I was referring to emptying your short term memory so that unwanted thoughts don't disturb your thinking.

</comment></comment></comment><comment><USER21> Sometimes willpower is the only thing you have when others say you don't.</comment>


Paul Buchheit: We all have tunnel vision - <AUTHOR> http://paulbuchheit.blogspot.com/2007/11/we-all-have-tunnel-vision.html

<comment><USER2> Suggestion: stay the fuck away from reddit, at least anything besides the science and programming subreddits.

Even frequenting programming.reddit you might get the idea that functional programming is poised to break into the mainstream any day now. Don't hold your breath.

<comment><USER3> "Even frequenting programming.reddit you might get the idea that functional programming is poised to break into the mainstream any day now. Don't hold your breath."

I think you're wrong on this one. In the Perl community (a mainstream community if ever there was one), Haskell is very popular, and a rather large swath of interesting work around Perl is in functional techniques and tools. Functional programming is definitely going mainstream.

Not to mention that Simon Peyton-Jones works for Microsoft and F# is getting quite a push.

<comment><USER4> My bet is to see someone "steal" a lot of FP ideas and integrate them into something aimed at being a more mainstream language, rather than Haskell or something of its ilk becoming popular. That seems to be the trend with things like Lisp and Smalltalk.

</comment></comment></comment><comment><USER5> one way to avoid this is deliberately trying to find non-corroborative evidence for every idea/opinion you form.

it's talked about in the Black Swan and is surprisingly difficult to do.

<comment><USER6> I actually have no idea what "non-corroborative evidence for an idea" means. Does that mean try to prove yourself wrong? Or to try to find evidence for the idea that, due to its difference from your original evidence, doesn't actually corroborate? Or ...?

<comment><USER7> "I actually have no idea what "non-corroborative evidence for an idea" means."

It's the opposite of <http://en.wikipedia.org/wiki/Corroborating_evidence>

<comment><USER8> not to be technical, but i believe "non" means "absense of," not "inverse of." so for a given statement there's a lot of non-corroborating evidence that is irrelevant

</comment></comment></comment></comment><comment><USER9> I think YC gives us tunnel vision too...

Maybe it's not such a bad thing, though.

</comment><comment><USER10> > _Supposedly people can hold about seven "items" in their mind at any one > time. I was never sure what that meant -- what qualifies as an "item"?_

This urban legend PB refers to actually originated with studies of short-term memory and remembering sequences of digits.

See [http://188.8.131.52/search?q=cache:PP_hzY7PqKgJ:www.knosof....](http://220.127.116.11/search?q=cache:PP_hzY7PqKgJ:www.knosof.co.uk/cbook/misart.pdf)</comment>


Migrating Gowalla Users (disambiguation, algorithms and magic trip itineraries) - <AUTHOR>

Friday, December 2nd at 1am (Europe time), after watching the always great "Bourne Ultimatum" film, I checked the last tweets before going to sleep. A non-surprising tweet announce that Facebook had bought Gowalla and they were going to close it.
They have a killing design and user experience, and it's a pity they couldn't do better. Even today when I look at their webpage or apps I can see a lot of small details that make me smile and say: 'they are so good'.  If you look at our homepage (http://www.touristeye.com), that comic was inspired by their design of 2 years ago. They also had a comic, and we thought a comic with 3 steps was a really good way to explain why TouristEye is the best travel app.
So, at 2am I started looking at their API to see how we can make the migration. It wasn't going to be an easy proccess, as Gowalla users do checkins in all kinds of spots; and we only want tourist-related places. I sent an email to our team asking for feedback. They said, yeah, weekend #hackathon! Saturday at 8am, already wake up and working on the integration :). At the end we have been all week with so many details. Problems we had:
* 1st problem. Matching Gowalla and TouristEye places to avoid duplicates. Glad we had already built a disambiguator to match more services. We have improved it this week testing with more data. It works as a servlet.
* 2nd problem. Choosing which spots to import. We are not going to import all kind of places, we are a travel guide. There are a lot of crap spots, that's the problem I personally have with Foursquare, Gowalla and all those services that have millions of places. So we decided to import some categories only. Then import the lists with more than 5 places, and import all the spots which are in those lists.
* 3rd problem. Could we create user trips automatically? Looking at the checkins of 2 years of a user, could we know which trips have he/she done? The answer had to be yes. And yes it is. It's not perfect, but after testing it with dozens of users, the algorithm seems to do a really amazing job. How we do it? Let me explain it so we can get more ideas and feedback:
- Let's get all cities where the Gowalla user has been, and have an array with all the days he has been.
- If he/she has been more than 20 days/year on a city, we consider it to be/has been his/her hometown. We do this to detect when people moves from one city to another.
- Now we remove all cities which are less than 37 miles (60km) from the hometowns, as they are their daily places, and TouristEye is just about travel.
- Finally we break the cities intro trips. There are currently 2 ways to know it's a different trip. If the user visits one if his hometowns in the middle, or if he hasn't done a checkin in 4 days and the cities on the sides of those 4 days are not the same (to prevent cases where the user hasn't checkin but it's still traveling on the same city).
- We are ready to create the trips for him/her, with all the days planned, and the places he/she has visited. You can view an example of a trip here: http://www.touristeye.com/Trip-NYC-t-34813/planning
I encourage you to migrate if you have been a Gowalla user. We'll email you when the script finishes. Please, tell us if you are or you are not happy with the result so we can take into account more cases and have a cleaner migration!
You can migrate here: http://www.touristeye.com/gowalla
Also, who wants to create his trips from his Foursquare account? Now it's just one more step ;) <comment><AUTHOR> You can migrate here: <http://www.touristeye.com/gowalla></comment>


Is HTTP Public Key Pinning Dead? - <AUTHOR> https://blog.qualys.com/ssllabs/2016/09/06/is-http-public-key-pinning-dead <comment><USER2> I figured interesting words would probably start to trickle out about HPKP after the last month or so of talks. Opinions are split on how much damage RansomPKP and other hostile pinning attacks can do. I'm inclined to believe that it's a big enough risk to warrant some action (such as more diligence by free certificate authorities -- the crux of the talk was rekeying at up to 20x per week enabling the attack). Others, notably Mozilla, are taking the wait- and-see approach. Google and Let's Encrypt were the most resistant to changing anything at all.

Anyway, Scott Helme's writeup did a quick and solid job of summarizing a lot of it. It's linked from the Qualys piece, but here it is if you'd rather get to it directly: [https://scotthelme.co.uk/using-security-features-to-do- bad-t...](https://scotthelme.co.uk/using-security-features-to-do-bad-things/)

Full Disclosure: I and buu700 ([https://news.ycombinator.com/user?id=buu700](https://news.ycombinator.com/user?id=buu700)) did the AppSec Glory talk at Black Hat / DEF CON and developed the RansomPKP attack pattern.

Edit for Fuller Disclosure plus added thoughts: I'd rather not see HPKP die because we've done some cool stuff with it over at Cyph. I'd rather see HPKP be refined with extra requirements, possibly some sort of tie-in with CAA (that little-known standard for enforcing which CAs can issue certs for your domain).

Speaking of CAA, we did a show of hands in both the BlackHat and DEF CON versions of the AppSec Glory talk. Five hands across thousands of attendees. That's it.

<comment><USER3> > CAA (that little-known standard for enforcing which CAs can issue certs for > your domain)

I'm sure the CA's absolutely _LOVE_ this - customers are signaling that they will stay with CA 'X'.

Given 'vendor lockin' or 'guaranteed recurring revenue', cert prices would drop thru the floor for 1st year only for renewals to increase.

<comment><USER4> CAA is just a DNS record that can be changed at any time, so there's no vendor lock-in.

CAs that implement CAA request that record and check whether they're permitted to issue the certificate; if they're not, they should refuse issuance.

<comment><USER3> 1) CAA tries to address the threat of attacker getting a cert issued for your domain - to carry out such an attack inherently requires taking over your DNS record or your domain account - if they can do this then they can also reconfigure your CAA record.

2) CAA requires the CA to perform additional operation (retrieve and check that DNS record) - not all of the 300-600+ CA's will do this - all it takes is 1 CA and the attacker will get his fraudulent cert from that CA.

<comment><USER5> >CAA requires the CA to perform additional operation (retrieve and check that DNS record) - not all of the 300-600+ CA's will do this - all it takes is 1 CA and the attacker will get his fraudulent cert from that CA.

CAA seems like another good step that the browsers can demand after a CA fucks up - like google has mandated that some CAs publish Certificate Transparency logs after they've mis-issued a cert, they should also be requiring CAA on threat of being revoked from the browser's trust lists.

<comment><USER3> Yeah - in theory, but in reality the browsers are reluctant because of all the other sites that a CA has issued certs for - no browser wants to be singled out for blocking some other non-related sites.

</comment></comment></comment></comment></comment></comment><comment><USER6> This is going to sound glib, but here goes:

1\. The fact that HPKP can create this kind of gigantic foot-gun is a pretty good sign that it actually works. So much of security technology turns out to be cosmetic. The stuff that isn't often is hazardous.

2\. Don't HPKP your blog. Don't HPKP your cat-sharing starting. Key pinning is a good thing, but it doesn't need to be universally deployed.

3\. In order to hijack your site with HPKP, an attacker needs to take control of your server; they probably need RCE. In other words: if you can't safely HPKP your secure messaging system (cough), you probably aren't yet qualified to be running it: something worse happened to your users than HPKP misconfiguration.

If there's a capability we're missing right now, it's probably the ability for sites with low security requirements to opt out of HPKP. Remember, the threat model for HPKP isn't fraudsters or trolls; it's state-level adversaries.

<comment><USER3> > 2\. Don't HPKP your blog. Don't HPKP your cat-sharing starting. Key pinning > is a good thing, but it doesn't need to be universally deployed.

Just to be clear, this is equivalent to saying don't bother with SSL for your blog or cat-sharing site (because without pinning someone else can get a cert for your blog/cat-sharing site).

> the ability for sites with low security requirements to opt out of HPKP.

opt out?! It's off by default.

> Remember, the threat model for HPKP isn't fraudsters or trolls; it's state- > level adversaries.

Again, by saying HPKP is only for high security sites (now you're saying only for those threatened by state-level adversaries) you are essentially saying SSL is useless for everything less.

<comment><USER7> > > the ability for sites with low security requirements to opt out of HPKP.

> opt out?! It's off by default.

I think tptacek meant that a site owner should be able to explicitly declare that nobody should be able to pin a key, so that nobody can hijack the site in the future.

<comment><USER8> An opt-in would make more sense, since people using HPKP are those that know the most about it. Just like an opt-out, it would have to be via a different channel than HTTP headers.

</comment></comment></comment></comment><comment><USER9> HPKP is a workaround, not a solution, to the problem of trusting every CA in the world to certify every site in the world — it's a symptom of the fact that the economics of CAs are completely, totally and fundamentally out-of-whack: relying parties have no relationship with the CAs they trust.

There are two things a CA can do: it can a) certify that a key is associated with some real-world identity or b) certify that a key is associated with some DNS name. The first is actually pretty desirable, but … there is no global namespace of real-world identities, so there's nothing a CA CertCorp can certify other than 'public key 0x1234ABCD0987FEDB is associated with CertCorp identity X,' which is … not nearly as desirable, or useful.

The second possibility is actually pretty useful, but also imperfect. To a huge extent, Amazon really is amazon.com; Google really is google.com. But, of course, .com could always decide to reallocate hostnames at will, regardless of its contracts (there may be contractual, civil or criminal remedies, of course).

Where does HPKP come into all this? It really doesn't: it's a half-assed way for a site to say, 'you visited me once, and trusted CA X to verify my identity then; in the future, please trust CAs X & Y' — but it does nothing at all to actually ensure that CA X was reliable in the first place! After all, CA X could have issued a certificate to some malfactor, who served the HPKP instructions to only trust X & Y.

It's smoke and mirrors, all the way down.

<comment><USER4> There's nothing stopping you from pinning exclusively to keys under your control. A pin counts as satisfied if the hash matches at least one of the keys in the certificate chain, including the end-entity certificate itself. In other words, if you do not trust _any_ CA, HPKP can still help you.

For most sites, pinning to a small set of CAs is sufficient and a vast improvement, but if you chose to trust no one, that's fine too.

</comment></comment><comment><USER10> Here's an issue not mentioned in the article, which gives me the half-baked feeling about HPKP:

Thought experiment: (just using companies I think HN will be familiar with, I'm not involved with these.)

1\. You're Tumblr or Basecamp or Slack with thousands/millions of user.example.com user subdomains

2\. You want to protect all of them with a root HPKP policy with includeSubdomains, because users will be bouncing between them, and with that many trust-on-first-use gaps, it'd be pointless otherwise.

3\. And then you want status.example.com hosted at StatusPage, blog.example.com hosted at Medium, and someapp.example.com hosted via GitHub Pages with a CNAME. Assume there are long-lived links to all of these subdomain URLs.

You can even assume that the third parties implement HPKP, and that they'll never have a hiccup.

There's no way to make this work without either:

A) Removing includeSubdomains, and no longer catch a large amount of cases as users traverse your site, barely solving a problem anymore

B) Serving a concatenation of every third party pin, opening up gaps huge enough where suddenly every CA is now authorized again, not solving a problem at all.

C) Noticing something I haven't.

And note that there are security (not merely vanity) reasons for using subdomains instead of paths for hosting user-generated content. I don't think this is too crazy of a scenario.

<comment><USER11> That's fairly easy to fix by putting your users on user.fooapp.com and your corporate pages on *.foo.com, just like you're already (for a different reason) putting static resources on foo-static.com.

<comment><USER10> I think a plumbing-level security implementation detail should be a little less intrusive than having to move either the corporate domain or millions of user's addresses. I can't imagine Tumblr rolling out HPKP now with this workaround (though github did do the .com -> .io thing, so who knows).

The spec should've just included excludeSubDomains=

<comment><USER11> For security, especially at the plumbing-level, simplicity is a virtue.

</comment></comment></comment></comment><comment><USER12> _EDIT: I was mistaken about something fundamental in how TLS operates. Unchanged question included below:_

I have an amateurish question. Please correct me with counterpoints if I'm mistaken.

In an attempt to 'explain like I'm five', HPKP adds an additional checksum, so that the server can say, 'if you see any certs whose Subject-Public-Key-Info doesn't match this checksum, _those aren 't me_, because I generated this checksum off of my cert's Subject-Public-Key-Info.

But since the Subject-Public-Key-Info is (by definiton) public, a CA _could_ issue an owner-unsolicited cert to the same site, containing the same Subject- Public-Key-Info, and which will pass pin validation and allow the site to be impersonated [1]. This may happen if the CA is rogue, or if the CA makes a mistake and issues a certificate to some bad actor other than the actual site operator.

So if HPKP is designed to protect against CA errors (malicious or accidental), how can it do that when it can't protect against CA errors, malicious or accidental?

[1] [https://tools.ietf.org/html/rfc7469#section-4.5](https://tools.ietf.org/html/rfc7469#section-4.5)

<comment><USER13> If the attacker doesn't have the private key that matches the Subject-Public- Key-Info, then he won't be able to impersonate your server given such a cert. (If he has, then there's no need to produce a replacement cert; the official one is good enough).

</comment></comment><comment><USER14> the issue is that HPKP is as user friendly as GPG but without the same benefits. (learning-curve vs ROI) Also IMO the underlying problem is choosing a CA's in the first place. None of them have 'Skin in the Game'[0]. How do you actually decide on which CA is trustworthy?? Trust is measured by how long they have been in the business and how many trust related incidents[0] they had. And how is that confidence measured/created? (hint it isn't Math but whoever has the best marketing or lowest price - not the most trustworthy certs).

CAs are a business model impatiently waiting to be replaced (with something decentralized). It's a self-serving quasi-monopoly that all other vendors that want to implement trust must use.

[0] [http://www.fooledbyrandomness.com/SITG.html](http://www.fooledbyrandomness.com/SITG.html)

[1] [https://www.schrauger.com/the-story-of-how-wosign-gave-me- an...](https://www.schrauger.com/the-story-of-how-wosign-gave-me-an-ssl- certificate-for-github-com)

<comment><USER14> afterthought: I wouldn't actually call HPKP dead. It's a prerequisite if you want to do any PKI related functions on your site. E.g. <keygen> support was dropped by most browsers and the industry is heading towards native clients. Nevertheless WebCrypto[0] is supposed to offer the same type of quality and you need both HSTS/HPKP if you want to secure your implementation in any meaningful way.

[0] [https://www.w3.org/TR/WebCryptoAPI/](https://www.w3.org/TR/WebCryptoAPI/)

</comment></comment><comment><USER15> May I ask a stupid question? Last time I heard about HPKP was as a solution to prevent an attacker from installing a fake root CA certificate in the _client system_ (a mobile device), so that it could observe the behavior of a mobile app, by posing as its API server (for any purpose of reverse-engineering). Is such an attack possible, and HPKP a reasonable solution to it?

<comment><USER16> Locally trusted CA certificates circumvent HPKP protection, because many organisations use TLS MITM attack to spy on their staff and browsers must support it in order to work properly in those environments. But it makes sense for browser. If you are making your own application, you can implement it in any way. Simplest way would be to embed public key in the application and check it without HPKP. Though, probably, if your app is popular, you'll face that in some networks your connections are MITMed and you (or user) can't do anything about it, so it's choice about dropping security or not to work at all.

<comment><USER15> > Simplest way would be to embed public key in the application

Yes, but one eventually faces the need to update the certificate on the server, and still support installed applications (which fails when apps have an obsolete certificate pinned). Hence the need for smooth certificate updates, and the good smell of HPKP. But it addresses another need, hence my confusion.

</comment></comment></comment><comment><USER17> DANE certificate pinning seems to like it solves most issues HPKP does, without the imminent danger.

It won't stop a dedicated attacker from MITMing a connection if they can block DNSSEC. However, if someone uses false BGP route to hijack your IP, they can't reach your DNS queries.

You'd still be fucked if you connect to a local wifi connection though.

<comment><USER6> The downsides of DANE are substantial:

* DANE is effectively a key escrow system. COM's keys would be held by the USG.

* In the same vein: people with cosmetic TLDs like .LY would be forced to concede their TLS policy to governments like Libya. Similarly: the Internet would be conceding that protocols should work differently and be less trustworthy depending on what country you're in.

* The "last mile" on DANE is spoofable, because DNSSEC doesn't protect the link between browsers and DNS servers; it's a "server-to-server" protocol. In particular: your ISP could still override pins!

* DANE doesn't actually work in practice, because of all the networks that block odd-looking DNS lookups; this is Adam Langley's point from "Why Not DANE In Browsers".

* Stale or misconfigured DNSSEC signatures knock whole sites off the Internet with no diagnostics. Unlike a broken HPKP pin, there's no UX saying something's wrong with the site; it just doesn't exist anymore. This is what happened to HBO Now on Comcast, and why Comcast users occasionally can't reach .GOV sites (which mandate DNSSEC).

* DNSSEC is far, far more expensive to deploy than HPKP (HPKP is just a header you add to your web server configuration).

* The cryptography behind DANE is 1990s-grade. The DNSSEC roots are still RSA-1024! (Don't worry! They've got the upgrade to RSA-2048 on the calendar!)

<comment><USER18> > * DANE is effectively a key escrow system. COM's keys would be held by the > USG.

They can already temporarily change DNS records and get certificate from any legitimate CA.

> * In the same vein: people with cosmetic TLDs like .LY would be forced to > concede their TLS policy to governments like Libya.

They already are doing this, see first point.

> * The "last mile" on DANE is spoofable, because DNSSEC doesn't protect the > link between browsers and DNS servers; it's a "server-to-server" protocol. > In particular: your ISP could still override pins!

Run a local resolver. If you have trusted server some you can use DNS over TLS.

* DANE doesn't actually work in practice, because of all the networks that block odd-looking DNS lookups; this is Adam Langley's point from "Why Not DANE In Browsers".

Then these networks are bad. For what reason network should be interested with more than UDP/TCP layer in DNS packet? Again, you can use DNS over TLS.

For other readers, look at discussion in [https://news.ycombinator.com/item?id=12382752](https://news.ycombinator.com/item?id=12382752)

<comment><USER6> Regarding your first point: no, they can't. See what happens if they try to exploit the DNS to get a Google certificate. (Notice the last several years of Google punishing CAs).

It is very important to DNSSEC advocates to convince people that our security can't be any better than the security of the .COM zone, which is controlled by the US Government through Verisign. But nobody accepts that level of security today. Arguments that we can escrow all our security with Verisign are DOA, post Snowden. Move on.

Regarding the rest: you can't persuasively rebut arguments by conceding they're correct.

<comment><USER9> > Regarding your first point: no, they can't. See what happens if they try to > exploit the DNS to get a Google certificate. (Notice the last several years > of Google punishing CAs).

If the U.S. government, or the Russian government, or the Luxembourgian government orders a CA subject to it to produce a certificate for a given key and hostname, it will be given a certificate for that key and hostname.

If the U.S. government, or the Russian government, or the Luxembourgian government orders a DNS provider subject to it to resolve all requests for a domain to a different IP than it would have otherwise, those requests will be so resolved.

If the U.S. government, or the Russian government, or the Luxembourgian government orders a network subject to it to route all traffic for one IP to a different IP it controls, that traffic will be so routed.

> It is very important to DNSSEC advocates to convince people that our > security can't be any better than the security of the .COM zone, which is > controlled by the US Government through Verisign. But nobody accepts that > level of security today.

Everybody accepts that level of security today, because that's the level of security we have today.

Actually, it's _better than_ the level of security we have today, because at least a DNS-level CA system would mean that no-one _other than_ someone who can compel .com can take over a .com hostname. Right now any government in the world can do so, at the drop of a hat.

Restricting that to the government a company or person is subject to greatly reduces the attack surface.

<comment><USER6> If the US government orders a CA to mint a bogus certificate for Google Mail, they will get the certificate, and then the CA will get put out of business, one way or the other.

There are consequences to suborning the TLS CA system. DNSSEC advocates pretend there aren't, because they have to, or their arguments become untenable, because there cannot plausibly be similar consequences for suborning the DNS: we know for a fact, with repeated existence proofs, that governments will manipulate the DNS (like DOJ did with file sharing sites), and we know .COM isn't going anywhere.

DNSSEC advocates will grudgingly acknowledge one half of the modern CA system security model: key continuity (pinning).

What they seem unable to acknowledge is the other half. To use Moxie Marlinspike's terminology, which he coined in an argument against DNSSEC: trust agility.

<comment><USER17> > If the US government orders a CA to mint a bogus certificate for Google > Mail, they will get the certificate, and then the CA will get put out of > business, one way or the other.

Probably, but in the year (or more considering that startSSL is still trusted) until the CA's trust is removed, that certificate still works.

Besides, if you accept that governments manipulate DNS, why do you deny they manipulate DNS to get domain validated certificates? Cause if not, we are back at trusting verisign for all websites that aren't statically pinned. (with that added 'bonus' that those statically pinned sites can be wrecked by their pin at any time)

</comment></comment></comment></comment></comment></comment></comment><comment><USER16> I won't use or recommend HPKP until there'll be a way to recover domain with lost keys. I think of something like bit in the certificate which will override HPKP and this bit should be very thoroughly verified by issuer, something like EV. Risk of losing domain is unacceptable and overweight security gains.

<comment><USER19> Well, there's an idea: require a special certificate (via an extension) to enable pinning.

</comment></comment><comment><USER20> I recently had an idea to make HTTPS more secure for the average user: What if Google were to check the certificates when crawling a site, ensuring that its keys did not change in a malicious way.

Now, sites could present a different cert to Google IPs and to users. To solve this problem, the html anchor element could be augmented with the certificates fingerprint. The users browser would make note of the fingerprint and alert the user if something is off.

I think embedding information about a certs finger print or its root CA in links makes a lot of sense, as it would allow a sort of "web of trust" between linked sites.

<comment><USER21> I get the idea, but it sounds like it would make the problem of link-rot 100x worse, and the number of broken links would quickly desensitise users to the warning.

<comment><USER22> Obviously it'd be optional, and it'd only make sense for dynamically-serving systems like a web search engine.

 _And_ a multi-domain'ed serving system. Like a blog host. It could serve links to other blogs it hosts (on completely different domains) with cert hashes, and suddenly make it very hard to MITM any of the served sites without MITM'ing all of them.

</comment></comment></comment><comment><USER4> The article makes some good points, but I don't believe the risk of malicious pins has any effect on adoption. It doesn't really matter whether your site currently uses HPKP or not - the attack works either way, as long as the user agent supports HPKP. Once an attacker has compromised your server in a way that allows them to add arbitrary HTTP headers, you're doomed.

That's not to say that things like RansomPKP aren't problems worth thinking about, but when it comes to adoption, time would probably better be spent improving tooling and documentation.

</comment><comment><USER23> Like tptacek said, "Don't HPKP your blog".

There's only a handful of sites that need HPKP and that's okay. That doesn't mean there is something wrong with HPKP. It's a useful tool for people with a specific threat model. If you are one of these sites then deploy HPKP and handle the operational responsibility that it comes with.

If you aren't one of these handful of sites then do not deploy HPKP. HPKP is a huge footgun and the benefits are not super large on an average case.

<comment><USER19> I think high profile sites are better served by static pinning, which seems simpler to maintain and also comes with the benefit of preloading. Thus, if you have those sites at one end, and lots of small/no-large-risk sites who don't need any pinning on the other end, you're left with no one for HPKP to serve in the middle. But, if HPKP were easier to deploy, perhaps more sites would use it.

</comment></comment><comment><USER24> > Or, if you're lucky, they seek ransom, giving you a chance to get the backup > pinning key from them and keep your business.

That's a rough choice -- let the business stumble/fail by moving to a new domain or accept the keypair from an untrusted party who's shown themselves to be untrustworthy. What's to stop them from sharing the private key with other ne'er-do-wells?

<comment><USER16> You can gradually migrate to another non-compromised key. Generate new key, add HPKP headers for both compromised and non-compromised key, keep it long enough so all users who got bad HPKP headers will get new ones, then replace certificate and set HPKP header to include only new keys.

</comment></comment><comment><USER25> The problem is if you dull HPKP, you potentially weaken the benefit that the really high-profile targets gain and therefore make it less useful for them.

Perhaps the answer is only a small number of websites actually need/want this functionality - and that's okay? Simply dulling it to broaden take-up may have a net negative affect.

</comment><comment><USER26> The article did mention this but I will re-iterate that mobile apps can benefit a lot from key pinning. That's a huge chunk of total internet activity. So, not that dead if you ask me.

The scenario of attacker having root(?) access to the machine warrants much bigger (or equal) concern than bricking your website in my opinion.

<comment><USER19> Just to clarify: in my article I discuss mostly HPKP as defined in RFC 7469. I think static pinning works great (and also has the benefit of being preloaded). Mobile and other types of pinning where you control both sides of a conversation also work well and are easier to get right.

</comment></comment><comment><USER27> Wouldn't a new record in DNS a safe dynamic way to publish the certs signatures?

<comment><USER28> It's too easy for the same attacker to MITM both the certificate and the DNS.

</comment></comment><comment><USER29> He says "bricking your website" while he means "your website will appear as if it has an invalid certificate".

Bricking means to crash something beyond repair.

<comment><USER19> No, when HPKP breaks your site no longer works, period. The error page doesn't allow clicking through. Some browsers (e.g., Chrome) support manual editing of the HPKP configuration so some users might be able to get around the problem, but that's unlikely to work for many.

Try it here: [https://pinning-test.badssl.com](https://pinning- test.badssl.com)</comment></comment>


NNCP: utilities simplifying secure store-and-forward files and mail exchanging - <AUTHOR> http://www.nncpgo.org/ <comment><USER2> "UUCP can be setup rather easily with few configuration files and few lines in each of them. But you have to _add some encryption and authentication overlay_ for securing you data transmission."

Any projects that attempted to do this sort of overlay for UUCP?

Incidentally, I like the way the author presents this project. Usage examples, thoughtful use cases, comparisons, protocol description and _packet formats_.

No hype.

<comment><USER3> I've never done this with UUCP, but if you don't care about anonymity you could do two-way authenticated TLS (both client and server have and present certs).

Maybe you could do a ratcheting protocol like Signal does, instead, which might be better.

There's a couple ways to do this that might be sensible depending on whatever constraints there might be in a deployment.

<comment><AUTHOR> Anonymity is not the goal, because it is another separate task.

NNCP contains nncp-daemon and nncp-caller that work with online protocol. It uses Noise protocol: [http://noiseprotocol.org/](http://noiseprotocol.org/) (Noise-IK). So ratcheting as Signal does is already applied. [http://www.nncpgo.org/Sync.html](http://www.nncpgo.org/Sync.html)

<comment><USER3> Wonderful, I had no idea what Noise was, or Signal for that matter, or NNCP despite reading the article. Thanks for the links! Guess I'll get back to my punch cards.

</comment></comment></comment></comment><comment><USER4> Related: _An off-grid social network_ (3 weeks ago) | [https://news.ycombinator.com/item?id=14050049](https://news.ycombinator.com/item?id=14050049)

 _> Scuttlebutt is [...] a peer-to-peer system ideal for social graphs, identity and messaging._

<comment><AUTHOR> Really interesting project! Must look on it much more closer! Thanks for the pointer.

</comment></comment><comment><USER5> This seems like a really interesting project. I've been recently playing with trying to get something like nntp to work over point-to-point HAM connections. This would solve a lot of the end goals I've been wanting to get to!

<comment><AUTHOR> Unfortunately NNTP(like) is the only thing I have never touched at all. I have several years of experience with FidoNet's echomail (this is like NNTP), but today I see that most of discussion is done in mailing lists and I have got no usecases I really could use with something like NNTP (and that is disappointing, because I really like that echomail/news nature).

</comment></comment><comment><USER6> I couldn't find your github mirror. (404, [https://github.com/stargrave/nncp](https://github.com/stargrave/nncp))

<comment><AUTHOR> Thanks for mentioning that. I forgot to remove that link. There is no mirror on Github anymore.

</comment></comment><comment><USER7> So that you're aware, this is nearly unreadable in Firefox on Linux: for some reason all the links are turquoise on pale blue rather than dark blue on pale blue.

</comment><comment><USER8> Should resubmit with a "Show HN: " prepended.

<comment><AUTHOR> Done in [https://news.ycombinator.com/item?id=14190970](https://news.ycombinator.com/item?id=14190970)</comment></comment>


Show HN: My First Site – MakerFile: Your Personal Maker Profile - <AUTHOR> https://makerfile.co

<comment><AUTHOR> OP here - to check out what profiles look like without signing up, here is mine: makerfile.co/jake</comment>


Ask PG: 1999-2005 - <AUTHOR>

I was just wondering what you during the time after you left Yahoo and before you started YCombinator?<p>(I know some of the things you did included writing essays and publishing Hackers and Painters, working on Arc, and deliver talks.) <comment><AUTHOR> I just read what I wrote (I must have been inebriated, sorry). Corrected:

What did you do during the time after you left Yahoo and before you started YCombinator?

(I know some of the things that you did included writing essays, working on Hackers and Painters, developing Arc, and delivering talks.)</comment>


Elon Musk Is Even Smarter Than We Thought - <AUTHOR> http://www.huffingtonpost.com/courtney-michaluk/elon-musk-is-even-smarter_b_5494114.html

<comment><USER2> Elon Musk's patent move is brilliant and inspiring, but this article's writing style is, for lack of a more fitting word, craptastic. There also seems to be zero useful additional information in it.</comment>


Tenerife Skunkworks: My road to Lisp - <AUTHOR> http://www.wagerlabs.com/blog/2005/02/my-road-to-lisp.html

<comment><USER2> _I considered Java too slow, Python too wordy_

OK, I was already feeling pretty leery of advice given by someone that rambles endlessly on about himself long before getting to the subject...but then we get this gem. Java is too slow, and Python is too wordy. Words fail.

<comment><USER3> That post is from 2005. Hopefully, I don't ramble about myself anymore!

</comment></comment><comment><USER4> This blog smells fake.

<comment><USER3> I don't think so ;-).</comment></comment>


Ensmallen – Flexible C++ library for efficient mathematical optimization - <AUTHOR> http://www.ensmallen.org <comment><USER2> I've written several frameworks for optimization and the approach here of defining special objective functions works quite well. I like the look of this and would love to try it out soon.

This is a small thing, but I'm really happy to see that their L-BFGS line search algorithm supports a maximum step size. Many numerical optimization libraries do not offer this simple feature, which is critical if your goal is to find a minimum that is close to the initial starting location. This is important in atomistic geometry optimization.

</comment><comment><USER3> In terms of c++ matrix libraries, I've found eigen3 to be better than armadillo (although armadillo is easier to use).

</comment><comment><USER4> Github is [https://github.com/mlpack/ensmallen](https://github.com/mlpack/ensmallen)

<comment><USER4> There's also a paper: [https://arxiv.org/abs/1810.09361](https://arxiv.org/abs/1810.09361)

</comment></comment><comment><USER5> This is looking very nice, can't wait to try it out. But I feel it's a missed opportunity to not provide some simple examples [1], as well as a benchmark to quickly estimate the amount of flops that can be expected. The main reason to use a library like this one over pyTorch or autograd/scipy is speed, after all.

[1] The test cases can be seen as examples, obviously, but they are not written in an easily accessible form IMHO.

EDIT: Looking more closely, it seems you need to provide an explicit gradient, no autodiff included, which makes it a complete nonstarter for me.

</comment><comment><USER6> Looked at the documentation, had to do a double take "Is this mlpack??? a fork??" then checked, yup, it's a header only version of the mlpack's optimization library. Very cool.

</comment><comment><USER7> Anyone know how this would compare to IPOPT for constrained, sparse nonlinear optimization?

</comment><comment><USER8> How does this differ from Ceres?

<comment><USER9> Ceres is great for sparse, structured non-linear-least-squares problems (that is, problems where levenberg-marquardt is applicable). I believe it was initially designed for bundle-adjustment problems arising from structure-from- motion applications.

Ceres also does work for dense problems and general convex optimization, but it doesn't provide as much of an advantage over other packages (that is, you could just call into any existing l-bfgs implementation and wouldn't benefit much).

For some other problems, Ceres doesn't scale or can get in the way. For example, if you have a single value to maximize and a large number of variables, then ceres' auto-diff system won't work (ceres::Jet is based on "forward-mode" automatic differentiation, storing partial derivatives on the stack. If you have a high-dimensional gradient, it can exhaust stack space). On the other hand, Ensmallen doesn't seem to come with any auto-diff functionality.

</comment></comment><comment><USER10> Sounds like the name is based on the perfectly cromulent word "embiggen"[1], but somewhat confusingly uses the first part of the word "enlarge".

[1] [https://www.merriam-webster.com/dictionary/embiggen](https://www.merriam- webster.com/dictionary/embiggen)

<comment><USER11> I don't think "emsmallen" can be said reasonably :)

<comment><USER10> Huh, turns out you're right, which is entirely why em- and en- both exist and mean the same thing.

[https://en.wiktionary.org/wiki/em-#Prefix](https://en.wiktionary.org/wiki/em-#Prefix)</comment></comment></comment>


Ask HN: State of NLP in JavaScript World? - <AUTHOR> I am thinking of creating a web application. The app will analyze the English text. It will need to run text-readability algorithms such as  https://en.wikipedia.org/wiki/Flesch%E2%80%93Kincaid_readability_tests.
The app will also need to find the syllable count of the words.
Are there reliable NLP JavaScript libraries (just like we've NLTK in Python) that you use in production?
Thanks in advance for all help. <comment><USER2> Tensorflow.js might fit your needs</comment>


FedEx leaves Amazon on the doorstep as the store becomes a competitor - <AUTHOR> https://arstechnica.com/tech-policy/2019/08/fedex-leaves-amazon-on-the-doorstep-as-the-store-becomes-a-competitor/ <comment><USER2> Lot of amazon hate out there these days.</comment>


Is Publishing Too Top-Heavy? - <AUTHOR> https://www.publishersweekly.com/pw/by-topic/industry-news/publisher-news/article/81637-is-publishing-too-top-heavy.html <comment><USER2> > "There used to be a lot more books that could sell 40,000–50,000 copies. Now > more sell fewer than 10,000 copies."

Well, _duh_.

Speaking as a working novelist (who has been earning a living that way for the past 19 years, thank you very much) I note that the editor quoted above was speaking of _print_ sales, not sales including ebooks. A _really prominent_ feature of the publishing landscape in the past decade has been the collapse of the mass market distribution channel -- small C-format paperbacks which are sold on 90 day credit: when payment is due the distributor must be paid in full or _receive proof of destruction_ in the form of stripped covers. (This channel was originally created for periodicals and expanded to books in the wake of the depression and the second world war. Trade books sometimes come in C-format -- especially in countries like the UK where the mass market channel crashed and burned in the 1990s -- but usually come in hardcover and A-format or B-format: larger paperbacks. These books are returned intact to the distributor for credit, and a lot of bookstores rotate their stock on a 90 day basis, relying on the wholesaler's line of credit to cover their expenses.)

Anyway: mass market paperback sales in the midlist (not the bestsellers: the other 90% of the trade) fell off a cliff once ebooks showed up. Ebooks were about 1% of my sales in 2005, rose to about 5% by 2008, then took off like a rocket: I'm now with a publisher who _prioritises_ ebooks (but issues hardcovers as well), with roughly 50-60% electronic sales. In contrast, the figures for my mass market paperback releases dropped from roughly 60,000 for my first novel, to under 10,000 a decade late ... although I'm earning about five times as much.

I have two hypotheses to explain this:

1\. MMPBs were traditionally bought as disposable reading: the vast majority of customers wanted a cheap read, and binned/donated/re-sold their MMPBs on the second hand market after one read. (Hardcovers, in contrast, tend to be collected by fans.) Ebooks fill the disposable reading niche _even better_ than MMPBs insofar as they're infinitely more portable, and these days you don't need a specialized gadget to read them: any modern smartphone will do.

(It's no coincidence that my rising ebook revenue took up the slack from my descending mass market paperback revenue.)

2\. Amazon, the 500kg gorilla in the ebook field, doesn't release any figures for direct sales of self-published ebooks, but I'm guessing they're huge. A lot of writers bypass the traditional publishers completely and go direct to ebook via Amazon, who don't offer the range of support services of a Big Five outfit but who also pass more of the money through to the author. (Disclaimer: I'm with two Big Five publishers. The higher royalties would be nice but I don't like having to recruit and pay my own editors and proofreaders and typesetter, and I want to spend my time writing rather than being a one-man publishing business. There's this thing called division of labour ...) Anyway: about 90% of the self-published material (my opinion is based on a three year deep dive into Kindle Unlimited) is rubbish that, frankly, isn't ready for publication ... another 9% is decent commercial-grade fiction, and 1% is stand-out star quality stuff, and _some of that is going bestseller under the radar_. That is: because it's sold on Amazon and AMZN doesn't feed numbers to Bookscan AIUI (Bookscan captures _retail_ sales at the EPOS terminal) they don't show up in the charts, even if they sell a million copies. And these books are invisibly eating away at the major publishers' market, corroding it below the waterline.

<comment><USER3> My take.

I have a huge collection of C format paperbacks from the 1960s and 1970s; as ephemeral as these are supposed to be, they held up really well until about 2005, by 2015 the yellowing started getting serious.

I (mostly) hate larger format paperbacks, since these are often poorly made -- you bring a $50 book home, open it, and the spine breaks right away.

The conventional wisdom that hardcovers are "better" than paperbacks seem to be belied by the fact that hardcovers are usually cheaper on the secondary market, sometimes by a little, sometimes by a lot.

<comment><USER2> The yellowing is due to acid leaching in the wood pulp paper; it _can_ be reversed or halted for a price, but it's accelerated in conditions of higher humidity.

US hardcovers are _supposed_ to be saddle-stitched and bound, and printed on acid-free paper. UK hardcovers use cheaper paper and perfect binding, as do paperbacks; "perfect" binding is basically cut sheets held together using a thermosetting glue. A lot of late 60s/early 70s UK paperbacks had such terrible glue that they were cracking and the pages falling out within a couple of years.

</comment></comment></comment><comment><USER4> Pro baseball teams don't measure the success of their player development systems by minor league ticket sales.

I guess its no wonder that book publishers are terrible at developing great authors.

</comment><comment><USER5> Publishers should be taking more risks. The risks make a list interesting.

Think PoC || GTFO bibles. My staff thought it was crazy to publish them. Haha.

<comment><USER2> I note that "No Starch" is a young insurgent publisher and not part of the Big Five.

The Big Five are _all_ about supply chain management contracts. That's all. Everything that can be outsourced is outsourced.

But this leaves them vulnerable to obsolescent boilerplate contracts horked up by an ossified legal department, which get in the way of business innovation.

Some of them are working around this, either by firing off contract updates for backlist titles to be signed by their authors or assignees: others by incubating internal startup ventures with from-scratch redesign of their workflow.

Example: Tor.com is an imprint of Macmillan, and its chief executive is Fritz Foy. It goes back about a decade and its front end is a very successful SF/F media website. Tor is _also_ an imprint of Macmillan, and its chief executive is Fritz Foy. It goes back about 40 years and its website is ... not its best aspect. In fact, Tor.com's entire editorial staff also wear Tor hats from time to time, although their marketing team are different. But the contracts might as well come from another star system. Tor's legalese focusses on the traditional trade fiction distribution channels and would you please sign this paragraph about microfiche and gramophone rights just in case we still use those formats, while Tor.com assumes ebooks have priority and by the way there'll be a hardcover issued via Tor as a distributor, and maybe a trade paperback.)

<comment><USER5> We've been around since 1994. :)

<comment><USER2> You're a young insurgent then!

Some of the big five have roots going back to _18_ 94, and earlier.

(Time runs at a different pace in publishing, if that makes sense.)</comment></comment></comment></comment>


An apology to pg and the latest YC class - <AUTHOR>

The other day I made a thoughtless comment, several actually, when I should have congratulated you all on the hard work you've put into your startups.  In the future, I will be more conscious and empathetic. Had it been me, I would have been offended by my comment, so I broke the golden rule.  Instead, I should have found something good to say, because I know there are plenty of good things I could have said.
I'm sorry and I wish you all great success.  I hope all your hard work is justly rewarded.
Sincerely,
fnid2 <comment><USER2> I looked up your comment since making this post. While I can agree that maybe criticizing these early stage startups might be a bit harsh, something about your comment resonated with me. There really seems to be a lack of _impressiveness_ with many of the YC apps, in terms of solving technically hard problems. Maybe that's just a CS student's bias.

<comment><USER3> isn't that true of most startups? There are a lot of business problems that can be solved by an (from a CS standpoint) uninteresting webapp. If you take one of those, you can focus on solving the business problem (a hard enough thing to get right by itself) without worrying too much about pushing the limits technically.

</comment></comment><comment><USER4> chalk this submission up as "should have been a comment"

</comment><comment><USER5> > Instead, I should have found something good to say, because I know there are > plenty of good things I could have said.

I disagree, criticism is much more useful than flattery.

<comment><USER6> "something good to say" is not necessarily flattery. It could be thoughtful positive feedback, which is just as useful as criticism (i.e. thoughtful negative feedback). Feedback can be useful whether it's positive or negative, what matters is the content:

                               Useful                 Not useful

      Positive feedback        "like x because..."    flattery
      Negative feedback        criticism              trolling

<comment><USER5> > It could be thoughtful positive feedback, which is just as useful as > criticism (i.e. thoughtful negative feedback).

I disagree, (supposedly thought) positive feedback is extremely common and rarely provides anything new (or own minds are well tunned to try to find reinforcing evidence to back our past decisions).

Negative feedback is much more rare due to social pressure to not 'offend' and other such bullshit, and even when it is not particularly thoughtful it adds something to our thinking as it is unlikely to be something we would have considered.</comment></comment></comment>


The Secret History of the Atom Bomb Is Now Available in Full Online - <AUTHOR> https://www.osti.gov/opennet/manhattan_district.jsp

<comment><USER2> [https://news.ycombinator.com/item?id=8210251](https://news.ycombinator.com/item?id=8210251)</comment>


Summary of all the MIT Introduction to Algorithms lectures - <AUTHOR> http://www.catonmat.net/blog/summary-of-mit-introduction-to-algorithms/

<comment><USER2> Awesome. I love your posts Peter, and have definitely echo'd some of your stuff directly into my .vimrc. Keep up the great work!

<comment><AUTHOR> Thank you! :)</comment></comment>


The Dating Ring Is Raising Money To Fly Women From NYC To SF - <AUTHOR> http://techcrunch.com/2014/03/04/the-dating-ring-is-raising-money-to-fly-women-from-nyc-to-sf-because-dating/

<comment><USER2> I thought it would be funny to make a joke site with this idea (even grabbed sadsingles.com a few days ago for it) - it's interesting someone is actually trying it.

The dating situation is so bad for single men in their early to mid twenties on the west coast that I'm starting to think it's worth bailing on the area entirely and moving to NYC just for that purpose. (Palo Alto has got to be magnitudes worse than SF which is still pretty bad).

At first it looked like the situation in NYC was even worse for women with the 150k vs. 50k difference, but the population of NYC is 8.2million where SF is only 805k. Still dramatic, but not nearly as bad.

<comment><USER3> I think the numbers are a red herring. Finding a _date_ is a numbers game. However, if you've spent any time in a committed relationship, you'll know that pretty much the only things that matter are how she feels about herself and how she feels about you (and the converses for you, which are heavily influenced by those). All the stuff about hotness, income, intelligence, background, careers, etc. fades into the background, and you figure out how to make it work.

I suspect the reason that dating sucks in the Bay Area is because the Bay Area attracts guys (and women, but to a lesser extent) who are overly concerned with how they're perceived by others. After all, everybody is chasing the latest hot startup that will make them wealthy and famous. That same insecurity - and the same insecurity that would make you bail on an area just because there are no girls - is massively unattractive to women.

I spent a year and a half dating around, including some very hot women. After realizing that none of that mattered and the only thing important was how I felt when I was with her, it took me all of 3 weeks to find a partner.

<comment><USER2> If dates are generally a prerequisite to relationships than the numbers issue tends to have a lot of influence (age does too).

My impression from living here isn't that it's people worried about their status, I think it's just that there aren't enough people - do you live in the bay area?

<comment><USER3> I do live in the Bay Area.

And my point is that there are enough people around that getting dates should not be a significant problem. I went out with about 25-30 people before finding my girlfriend; to get to that involved about 200 OKCupid messages. There were about 5 dates in the 3 weeks before and 3 weeks after I first met her.

</comment></comment></comment></comment><comment><USER4> This article makes it obvious how far behind technology NYC is compared to SF. Men in SF don't want women from NYC flown to see them. Men in SF want women delievered by Quadricopter drones.

</comment><comment> daniel-cussen I believe this is not the first time this happened in San Francisco.

During the Gold Rush, the gender ratio was about 50 men per woman. It got better with the years, but party organisers kept doing their damnedest to get women to go to their balls, inviting (and arranging for travel for) women from as far as Missouri, over the Pony Express.

</comment><comment><USER5> Pardon the language, but this reminds me of the Fuck Truck (at least in decades past, when the MIT male to female ratio was way more out of whack):

[http://en.wikipedia.org/wiki/Wellesley_College_Senate_bus](http://en.wikipedia.org/wiki/Wellesley_College_Senate_bus)

<comment><USER6> Sure, but given these people could not get a date in NY, I would suggest it is more like the Fright Flight, or the Plain-Jane Plane.

</comment></comment><comment><USER7> And in a third, different place on Earth a man is choosing between keeping his sanity or caving in and typing "valleywag.gawker.com" in his browser's address bar.

</comment><comment><USER8> "Also, not great, as pointed out by another TechCrunch writer: The implicit assumption that all dating is between women and men."

Another possibility is that there's no first-order[0] geographical arbitrage opportunity presented by gender imbalances when men seek men and women seek women.

[0] there may be "second-order" abritrage opportunity, in terms of sub-types of men or women.

</comment><comment><USER9> Humor is all this is.

Open question: where do men here in SF try to meet women?

As a man, I have yet to see such a lack of single women in the city that would necessitate a project like this, even when I was dating around for 9 months.

Me thinks either standards are too high or don't go far enough out of your comfort zone. I could imagine being wrapped up in a startup, eat, breathing, sleeping it. And yes, not too many single women in the startup scene.

But startup != San Francisco, though it may seem that way from the news media.

<comment><USER10> As a guy with wickedly amazing success on OKCupid and Craigslist in New York and elsewhere, in SF alone for a month, I can confirm that the usual methods that smart, hardworking, tech-savvy dudes use for finding strangers to date are completely oversaturated in SF.

Offline is the way, obviously, but ain't nobody got time for that.

</comment></comment><comment><USER11> They need to make this frictionless...much like Uber. Make an app where I push a button and a 'dateable' girl gets delivered as soon as possible to a bar of my choice for an instant date. She doesn't have to be from nyc.

<comment><USER12> LOL. :)

The meta: know what you really want. Sex, companionship or kids: select up to two adjacent goals. Pick all 3 if you have a second condo and a burner phone.

</comment></comment><comment><USER13> ...and why didn't they raise the money to fly a planeload of eligible MEN to NYC?

<comment><USER14> ...because like Horoscopes, Magazines telling us what we should eat/wear/say, and the rest, Men don't generally fall for this kind of thing.

<comment><USER15> Yet they're still funding this campaign to have women sent to them. Which is one of the saddest things I've typed in recent memory.

(and if you really think that "men" don't fall for being told what to eat/wear/say you're horribly, sadly mistaken)

<comment><USER16> Years ago, Dilbert had a hilarious series of strips about mail order brides. But I agree that it is sad.

I'll just leave this here.

[http://search.dilbert.com/search?p=Q&lbc=dilbert&uid=7339291...](http://search.dilbert.com/search?p=Q&lbc=dilbert&uid=733929174&ts=custom&w=Mail%20Order%20Bride&af=&method=and&view=list&filter=type%3acomic&isort=date+rev)

</comment></comment></comment></comment><comment><USER17> My issues with this: 1\. Whats the goal from the date? Nobody is going to commit to a long term relationship, and I doubt you will convince a guy to leave his job in SF to move to NY from this date.

2\. Waste of resources/emissions. Sure, the effect of this is negligible since the planes will be flying anyways, but in principal this just seems like a huge waste of resources.

3\. Given 1, it seems that this "date" is really just going to be mostly one- time affairs. Which is cool. But why not just find a date locally or hire an escort.

<comment><USER6> Not sure I agree about point three - why do you think they will get more success in SF than in NY? If they cannot get a "date" in NY, I am not sure they would get one in SF. Unless the organisers provide everyone with masks, cover of darkness and free booze!

</comment></comment><comment><USER18> Dating Ring couldn't wait until a single date's worth of Groupon signups were had before moving on to the next pivot? [http://www.groupon.com/deals/the- dating-ring-1](http://www.groupon.com/deals/the-dating-ring-1)

</comment><comment><USER19> The ratio in New York City is hardly a problem since the population is so big (8 million). It is like 53% women and 47% men.

I am a man and went to a college that was 55% men and 45% women. I never felt like there was a lack of women.

<comment> T-hawk The raw numbers don't paint an accurate picture of dating inequality. The pairwise nature means you need to subtract out those already paired with a dedicated partner. That will vary for different groups and can cause extreme imbalances in the dating population that you might not expect.

In your college of young adults, suppose 30% of the population is already paired, which is 15% of each gender. That leaves 40% available men to 30% available women, a 1.3:1 ratio, not too bad.

In New York City's adult population of all ages, suppose 86% of the population is already paired, 43% of each gender. That leaves 10% available women chasing 4% available men, a 2.5:1 ratio, quite lopsided.

</comment></comment><comment><USER20> VCs should be concerned. They are trying to fly in distractions. What if your founder starts wasting time dating women? Shouldn't they be focused on their startups? Startups are on a peninsula for a reason :)

</comment><comment><USER21> Link to the crowdtilt: [https://www.crowdtilt.com/campaigns/working-on- it/descriptio...](https://www.crowdtilt.com/campaigns/working-on- it/description)

</comment><comment><USER22> Most expensive city in America, the most downright tyrannical laws in the country, might come crashing down on your head at any moment... oh, and NO WOMEN.

I thought you guys were supposed to be smart.

</comment><comment><USER23> I wonder what proportion would be willing to relocate either direction, or is that not the idea? Long distance never worked out for me.

</comment><comment><USER14> Wait - so they can't get a date in NY, and they think the standards are lower in SF, so they might get a date?

<comment><USER24> Majority opinion is that NY nightlife/dating life has higher ratio of girls to guys while SF has way more guys. So it doesn't mean standards are supposed to be lower, rather the ratio should even out between the 2 cities.

<comment><USER12> It's the "elsewhere is valuable" "logic."

Protip for men: Davis, CA not during summer has the highest concentration of unmarried, college educated, single women of any college town in the US, outside the Boston metro. Thursday night is salsa night at The Grad. Get your silly colored sunglasses and go.

The other trick is to have women fly out to meet you, oh wait, that one has an API now. :) Seriously, distance is another "barrier" so it is good for the stationary party (not that kind, the human kind).

<comment><USER4> > The other trick is to have women fly out to meet you, oh wait, that one has > an API now.

curl -X POST [https://thedatingring.com/api/v1/women](https://thedatingring.com/api/v1/women) -d '{"hair": "brown", "interests": ["me, duh", "froyo", "kale"]}'

=> {"flightNumber": 1525, "airline": "United", "departureTime": 201403051250, "arrivalTime": 201403051545}

<comment><USER16> Bad cert.

<comment><USER12> Did she lie about that too or was she using GnuTLS?

</comment></comment></comment></comment></comment></comment><comment><USER12> "It's like Great Expectations but with airplanes."

Any service that takes money from people by preying upon their dreams is inherently evil. This doesn't solve dating because it's unsolvable on purpose. Unplanned spontaneous clicking probably won't happen when it's the advertised goal. Joining interest groups on meetup would be infinitely better.

<comment><USER25> Spontaneous clicking is more likely to happen if you meet a lot more people!

<comment><USER12> Much less likely if that's the advertised goal. It's better to go about group activities people enjoy, and not let dating become their all-consuming goal.</comment></comment></comment>


How I'm learning "Machine Learning" - <AUTHOR> https://medium.com/@harpermaddox/how-im-learning-machine-learning-a0f88538eba1#.8ykeifa4q <comment><USER2> I think I got to chapter 6 of deep learning(deeplearningbook.org) before I ran out of steam. I went non-traditional as a Chem then physics-pre med major before grad school in comp sci so my linear algebra is piecemeal from quantum physics and math methods in physics. I really wish I took linear algebra before going through my cs grad school, but now I"m almost tempted to do Gilbert Strang's course on OCW to brush up on it. Linear algebra is the key to this stuff--well some tensor analysis as well, which, in my case, Dr. Houston snuck into math methods using Einstein notation. Yes, I did the Tensor Flow examples with the MNIST data, which was neat. I'm going to try the Coursera course as suggested by this article. I have links to Strang's linear algebra as well as a Drexel University course on Tensors on my page ([https://hackereducation.wordpress.com/hacker-education- conte...](https://hackereducation.wordpress.com/hacker-education-content/))</comment>


Forget Alaska. North Korea Might Soon Be Able to Nuke New York - <AUTHOR> http://www.thedailybeast.com/forget-alaska-north-korea-might-soon-be-able-to-nuke-new-york <comment><USER2> It's bad for nuclear non-proliferation, but it seems like having nukes is some sort of exclusive club and not having nukes means other countries can bully you around.</comment>


Man camps in fracking shale going broke. Any ideas for repurposing the buildings - <AUTHOR> http://www.bloomberg.com/news/articles/2015-04-15/oil-s-new-boomtowns-head-toward-bust I work in both the Permian and eagleford running man camps. The manufacturers are going under quickly. Cheap mobile. temporary housing is available at low costs. <comment><AUTHOR> Many of these camps are really hurting financially. The major manufacturers are doing poorly as well. Any ideas on repurposing these mobile dorm like buildings. I work in both the Permian and eagleford.</comment>


Show HN: Graph-based data storage REPL in C - <AUTHOR> https://github.com/incrediblesound/Graph-Reply

<comment><USER2> Some general feedback:

In C you separate your implementation from your interface with header files. If you include .c files in .c files it will end up causing a lot of problems and start making includes order dependent (like how your code will break if you move #include "structures.c" down a few lines).

Generally you want a header file for each source file and put the stuff you want to share with other files in there (prototypes, typedefs, etc). For example, disk_utility.h would have prototypes for save_to_disk and read_from_disk and then instead of including disk_utility.c in your replay.c file you'd include disk_utility.h.

Without compiling the code I see a lot of things that are, at least, warnings. When you're learning you really want to compile with -Wall and pay attention to warnings.

It's a neat project and I hope you continue learning C, it's a great language.

<comment><AUTHOR> Thanks for your feedback! I spent a lot of time chasing down memory leaks with this project, which was fun, but clearly that's not the last of the hurdles with C. My dream is to program space robots, which probably will never happen, but I feel one step closer every time I build something in C.

<comment><USER3> I've made a pull request that contains a reasonable mount of feedback. I hope it's useful and instructive!

<comment><USER4> Those are great reading, thank you (even if I have nothing to do with the project :-). For others:

[https://github.com/incrediblesound/Graph- Reply/commits?autho...](https://github.com/incrediblesound/Graph- Reply/commits?author=LukeShu)

</comment></comment></comment></comment><comment><USER5> C is not a safe language sort of by definition, so you always have to consider all the possibilities. For example, you use sscanf in places, with fixed sized buffers. What happens if someone were to pass a lot of data into your program? What happens if the on-disk structure gets corrupted and a 'type' becomes more than 5 characters?</comment>