When should you store serialized objects in the database? (2010) - <AUTHOR> https://www.percona.com/blog/2010/01/21/when-should-you-store-serialized-objects-in-the-database/ <comment><USER2> Bad ideas from 5, 10, er 20, er 30 years ago are stil bad ideas.

I know the HN police will cite me for no citation, so I'd say it comes with experience. The law changed at one point, and we were legally bound to be able to locate a customers record by a piece of data in a blob. The only way to fix the problem was to dump the massive (1tb+) table and reinsert them into a real schema. The engineering effort to do this took 9 months to get it right, because other people changed the way blobs were written out over the course of years.

Being clever doesnt pay, again.

<comment><USER3> > Bad ideas from 5, 10, er 20, er 30 years ago are stil bad ideas.

I agree 100%.

I read that title and expected the post to begin with "Never. You should never serialize objects into a single field." I was disappointed.

If you need schemaless storage, use a schemaless DB. I don't understand what's so difficult about that. I wouldn't try to shove unstructured data into PostgreSQL any more than I'd try to shove relational data into MongoDB.

<comment><USER4> HAHA, I tend to agree that 'never' is the only good answer.

</comment></comment></comment><comment><USER5> I think it's rarely a good idea to store blobs of any kind in the database. I've seen systems that store pretty large files as blobs (even base64 encoded ones once), then do 'select *' on the table and wonder why their query performance is so terrible. Use a filesystem, that's what it's for.

For stuff like this then I would say it's always preferable to store a json encoded representation rather than a format like pickle (python's object serialization format). If you don't and some clever chap works out a way to write input to that field then you've got an easy RCE. Plus it's easier to debug JSON, and databases like PG have a native data type for it.

<comment><USER6> _it 's always preferable to store a json encoded representation_

That's just the format du jour. Ten years ago it would have been "store an XML encoded representation" and ten years before that it would have been some delimited representation. Tomorrow it may be yaml or something even more hideous.

Blobs in the DB can make sense in some situations but they should really be blobs: images, or other binary/raw data. But beyond a certain size it's almost always easier to manage things like these in a filesystem and just store a pointer (filename) in the DB.

<comment><USER7> But there are database tools emerging to deal with JSON, like Postgres (and Mongodb.) I haven't used Mongo in production, but postgres's json support is fantastic and materially better than anything I have seen with XML. Sometimes you really do need to store nearly schemaless data and operate on it for some time.

<comment><USER8> As reference there are/were tools in something like MSSQL for dealing with XML.

<comment><USER9> mysql has had xml/xpath query functions for a while too, iirc.

</comment></comment></comment></comment></comment><comment><USER10> I was on the DBA team at FB and I spent the better part of a year working on the deployment system for online schema change. It was a pain. Other companies have done quite a bit of work on this as well (Shift from Square, etc...).

Later on I joined Pinterest as their first MySQL DBA. They had copied the sharding system from FB, but instead of having a bunch of columns, they just stored a JSON blob. This saved them from learning how to perform schema change until I joined the company. This is a pretty incredible feature.

We have a new feature under development (which will be open sourced as part of Percona MySQL) which will allow column level compression with an optional predefined dictionary. During testing, this resulted in a 30% additional reduction in spaced consumed versus InnoDB page compression AND doubles our peak QPS at lower latency. This would not work well with many individuals columns, but kicks ass for JSON blobs.

[http://www.slideshare.net/denshikarasu/less-is-more-novel- ap...](http://www.slideshare.net/denshikarasu/less-is-more-novel-approaches- to-mysql-compression-for-modern-data-sets) (slides 37, 40, 41, 42)

</comment><comment><USER11> For anybody using sqlite, they have good documentation about this very question: [https://www.sqlite.org/intern-v-extern- blob.html](https://www.sqlite.org/intern-v-extern-blob.html)

<comment><USER12> Is this really the same thing? This page compares storing blobs in sqlite vs in a separate file.

I think a better sqlite page about the concept of serializing things in your database is the fact that sqlite has json support.

[https://www.sqlite.org/json1.html](https://www.sqlite.org/json1.html)

<comment><USER13> That's with a loadable JSON1 extension, which one won't find e.g. in Android.

Though checking now for this, someone has packaged a later version of SQLite with this extension [1]. I wonder if there is any possible performance advantage when using a system provided SQLite vs. one installed with the application?

[1] [https://github.com/requery/sqlite- android](https://github.com/requery/sqlite-android)

<comment><USER13> Note to self: should have read the whole project description, there's a performance chapter answering this exact question.

</comment></comment></comment></comment><comment><USER14> It makes sense to store serialized data structures in the database when these conditions apply:

1\. There are no use cases that would require you to SELECT on the fields in the serialized data structures.

2\. You anticipate that the data structures are going to change frequently during development, so that turning them into relations is going to involve a lot of schema migrations.

Basically you give up the possibility of being able to SELECT on some of the data in return for being able to change its format rapidly and cheaply.

I worked on a project recently where this was helpful -- when I designed the database schema I didn't know the details of many of the data structures that were going to have to be stored there. From the use cases I could deduce the set of fields that would need to be SELECTed on, but the other fields were ill-defined. By storing them as blobs (actually as JSONB fields, since this was PostgreSQL) I could safely defer the decision about how to design these parts of the database, without incurring lots of schema migrations along the way.

<comment><USER15> Be very careful with this logic. I'm in the middle of cleaning up a project where the original developer did this calculus and stored a lot of metadata as JSON strings in text fields. Now, three years out, we have new reporting requirements that need to be able to filter on some of that metadata.

Experience has taught me that it's next to impossible to know what types of queries will be needed for the entire lifetime of a project. If you absolutely must store data like this, please leave a clear migration path for moving some or all of it into well structured tables when (not if) it becomes necessary.

<comment><USER14> I understand your concern. Perhaps if I give an example, it will be clear why I followed the approach I did.

The project is an industrial control system. Different kinds of item are processed by the system. Each kind of item needs image acquisition parameters, for example field of view. There's no use case that needs to select kinds of item whose image capture parameters specify a particular field of view, and it's clear that such a use case is very unlikely to arise — the image acquisition parameters are part of the internal specification of the system, not something that any end user is interested in.

At database design time it wasn't settled which model of camera was going to be used, and so it was far from settled which parameters were going to be needed. Using a JSONB field for the image acquisition parameters meant that I could leave the specification of these parameters to the camera programmer, without having to incur a series of database migrations as the correct set of parameters were worked out.

Even in your project, where new reporting requirements eventually arose, it doesn't mean the original decision was necessarily a bad one. Sometimes it makes sense to incur technical debt in order to bring a product to market in a timely fashion.

</comment></comment></comment><comment><USER16> You can get away with it in Postgres. The app I work on stores phone numbers in a JSONB array in the following format:

        [{
          "tags": ["cell"],
          "number": "1231231234"
        }]

Here's a snippet demonstrating how you can do a lateral left join on the column to find the number tagged 'cell' in tags array:

        select * from mytable t
        left join lateral (
          select phone->'number' as cell_phone from
          jsonb_array_elements(t.phone_numbers) phone
          where phone->'tags' @> '["cell"]'
        ) p on true;

</comment><comment><USER17> I am working on a system where the data is serialized using Python's pickle and stored in the database. Absolute nightmare for debugging as its basically unreadable.

<comment><USER18> Can't you open a Python shell and unpickle it?

<comment><USER17> Gave me an error the last time I tried to that.

Plus even if it did work, it involves logging into the server, activating my python virtualenv, pulling the data out via the python / Django shell an unpickling and printing it. As opposed to running a query on my local machine connecting to the database. When you are debugging a problem and just want to get an overview of what is happening, that is a hell of a lot of hassle.

</comment></comment></comment><comment><USER19> PostgreSQL and jsonb: [https://www.postgresql.org/docs/current/static/datatype- json...](https://www.postgresql.org/docs/current/static/datatype-json.html)

<comment><USER20> Yes, maybe use a database that allows you the best of both worlds, a serialised blob that happens to be queryable and generally really high performance.

<comment><USER19> Queryable and even indexable! Hardly a trade-off at all, really.

</comment></comment></comment><comment><USER21> I was working on a massive CRUD project - hundreds of end-user customisable textarea fields. Despite this sounding like a project perfect for a NoSQL database, the in-house team that would be maintaining it were MySQL experts, and didn't want to support MongoDB or anything like that.

So, yep, we stored everything as serialized objects in the database. We had a separate table for 'change events', and whenever someone changed the contents of one of the textareas we stored it in that table. A worker would eventually update the the serialized object, but in the meantime, we would load the serialized object and apply all of the changes that had happened since it was last updated. Basically, an Event Sourcing pattern.

So it was either that, or the EAV route, or the 'end user altering the database' route. The latter two options sounded even worse. Our solution worked out pretty well. Admittedly, it only had a few hundred concurrent users, and even the biggest document was never more than 100K.

So, it can work, but YMMV.

</comment><comment><USER22> We serialized some XML as a backup to the data we were extracting (and properly modeling), given that the vendor was prone to changing the schema without proper notification, and that there were some data elements we weren't using at that time.

We also built the necessary tools to extract/re-process records easily, and the architecture worked well for us. As our needs or the schema changed we could easily accommodate those changes without undue effort.

It doesn't directly address the question, and I'm not sure that I'd use the same solution if the volume were predicted to be significantly higher, but in our case it worked beautifully. (Happily our volume was predictably within a known range, for reasons I won't go into.)

</comment><comment><USER23> (2010)

Uber moved to a similar architecture in 2014-2015, ~5 years after article and the original Friendfeed post. Being able to operate MySQL predictably at scale is extremely valuable to high-growth companies, enough to tilt in the favor of unconventional schema choices versus less proven NoSQL alternatives.

[https://eng.uber.com/schemaless-part-one/](https://eng.uber.com/schemaless- part-one/)

[https://eng.uber.com/schemaless-part-two/](https://eng.uber.com/schemaless- part-two/)

[https://eng.uber.com/schemaless-part-three/](https://eng.uber.com/schemaless- part-three/)

</comment><comment><USER24> In my experience, databases outlast the applications built on top of them, so it makes no sense to cut corners on the data modelling.

Except, of course, if the data only exists to support the application (some sort of buffer, cache, or session storage).

<comment><USER25> > databases outlast the applications built on top of them

sure but this needs to be balanced with performance UNTIL then

</comment></comment><comment><USER26> This is the consequence of the chosen programming language not being adapted to work with a relational database. The language is made to work with objects, and the access to DB is clumsy, trough library functions, which are deeply encapsulated, and the language and the DB are two different worlds. In SAP's ABAP language relational database access is integrated into the language. In SAP, when you create a database table, the structure of that table will be automatically available to any program as a structure datatype. So if you change a table definition, you will also change the data type used by the programs. Doing table changes is supported by a database tool that will automatically copy records from the old table to the new one if necessary. Its easy to find all references to a DB table, and recompile the sources. Its actually done automatically when both program and DB structure changes are deployed. Whatever change the developer does in the development system, that change will be automatically adjusted in the productive system on deployment. This makes any table structure change pretty easy, the development environment takes care of that. Wherever SAP applications use blobs to store data (for example HR payroll), those are the worst to develop with. Doing a non-simple change on a TB big table would surely cause disruption in a SAP system too, but other techniques are available for those cases.

</comment><comment><USER27> _" If the application really is schema-less and has a lot of optional parameters that do not appear in every record, serializing the data in one column can be a better idea than having many extra columns that are NULL."_

Why not just use a document oriented database instead? Seems like a good use case for MongoDB for example: [https://www.mongodb.com/compare/mongodb- mysql](https://www.mongodb.com/compare/mongodb-mysql)

<comment><USER28> Also the assumption is that you don't need to report on the data. If you need to report on the data then you might need to create index on those columns for performance which you can't do on a blob.

</comment></comment><comment><USER29> Agree, storing binaries in database is generally a bad idea. It would be a really miserable idea if it were being done without a sane persistence API. In the Python world, ZODB - [http://www.zodb.org/en/latest/](http://www.zodb.org/en/latest/) is tightly coupled with the language but works reasonably well in practice. The storage layer is pluggable and [https://pypi.python.org/pypi/RelStorage](https://pypi.python.org/pypi/RelStorage) provides storing pickles in RDBMS.

ZODB is arguable a novel approach to persistence using Python. And certainly worth taking some time just to play with it -- the barrier of entry low, e.g. `pip install`. But for each positive there are negatives..

"You got it buddy: the large print giveth, and the small print taketh away"

<comment><USER30> From my personal experience, I completely agree.

For an academic project, I was calculating ~2mln RNA-RNA interactions from their sequences. Since this calculation stays a requirement for all further calculations, being the naïve kid I was, I started pickle'ing the results.

To feel like the cool kid, I wanted to involve a database somehow -- so after trying out a bunch of options, I finally settled for ZODB. As the project scaled up, soon the ZODB started being a big pain, because as I recall, it only allows a limited number of connections even in the read operations.

Lesson learned, though, it now resides as a lookup table in a PgSQL instance.

</comment></comment><comment><AUTHOR> Link about FriendFeed (mentioned in the blog): [http://web.archive.org/web/20100314211658/http://bret.appspo...](http://web.archive.org/web/20100314211658/http://bret.appspot.com/entry/how- friendfeed-uses-mysql)

</comment><comment><USER12> What about this reason? What if your program is pretty much entirely used from a JSON REST service? What if these JSON objects need to also be sent between machines? What if they need to be exportable to files sometimes?

Now imagine the same program also has an internal database where these JSON objects can be imported and used. Does it make sense that, when actually in use, these objects are relational and split between 10 complicated tables? Why should someone bother writing complex import/export conversion functions, maintaining them in the future, and having worse performance. Wouldn't it be much simpler, maintainable, and faster to just plop the JSON in the database?

<comment><USER31> So, my question after storing xml in a database and using their xml features to build indexes is this. Are you looking for a database or a search engine? what will you gain from a database if you are not using it's features, over saving to disk and building a query index?

<comment><USER12> What you would gain is the ease, simplicity, and stability of using your favorite sql engine. You get ACID transactions and the ability to add and remove from large lists using low memory, for free.

</comment></comment></comment><comment><USER32> Proper databases have JSON or other serialized field types. Even mysql 5.7 has some support for this. There is no reason you should hit a limit on one of your tables at a few hundred thousand records because you're an idiot and stored a ton of serialized data in mysql. It happens all the time though.

</comment><comment><USER33> The answer is, of course, almost never.</comment>


WTFPL - <AUTHOR> http://en.wikipedia.org/wiki/Do_What_The_Fuck_You_Want_To_Public_License <comment><USER2> See also : The Romantic WTF Public License

[http://danslalune.posterous.com/the-romantic-wtf-public- lice...](http://danslalune.posterous.com/the-romantic-wtf-public-license)

It's even more free than the original since the license is released under itself. Released on Valentine's day by yours truly :-)

<comment><USER3> I would say that the postscript of that license is the one thing that I consider missing from the original WTFPL. If I'm giving away my software for free without restrictions, I sure as hell don't want to get sued over it.

</comment></comment><comment><USER4> I find it funny that this apparently meets the Wikipedia notability standards.

Anyway, I always found it strange that anyone would choose to use this license. The concept behind it is great, but why include unnecessary profanity in the license terms? It seems like it would only make it more difficult to defend the validity of the license and any works released under the license.

<comment><USER5> What would you need to defend?

<comment><USER6> Scenario: Alice writes a program, puts it under that licence, Bob uses the program, Alice dies and her heir Eve wants to get money from Bob for using the software. Eve tries to get a court to declare the licence invalid, so that Bob has to re-licence the program under new conditions or stop using it.

<comment><USER7> Assuming that it's possible at all to declare the WTFPL invalid (why would it be?) - he can re-licence the program to a licence giving everyone exactly the same rights. It is allowed by WTFPL.

<comment><USER8> Bob is not the copyright owner. He can't re-licence it to anything.

What he can do is re-license a right to use the program, under terms specified by the copyright owner.

I've attached extra meaning to c vs s in the word licence, but the definition you used is not the same as parent.

<comment><USER6> Yes. And if the new copyright owner Eve declares the original license invalid, Bob won't be able to re-licence (or even use the program any longer). Also Alice and Eve could be the same person who just changed her mind.

<comment><USER7> I don't think that's true. Eve can change the licence on her work. That doesn't affect any copy that Bob obtained beforehand. You cannot change the licence in a way that affects things already released - otherwise you'd be able to release something under BSD and after a year say "I'm changing the terms, new licence requires everyone to pay me X, even if you obtained the software on BSD terms".

Referring to the WTFPL author (<http://sam.zoy.org/wtfpl/>):

        Can't you change the wording? It's inappropriate / childish /
        not corporate-compliant.

        What the fuck is not clear in "DO WHAT THE FUCK YOU
        WANT TO"? If you do not like the license terms, just
        relicense the work under another license.

<comment><USER6> Yes, you can't normally do this --- as long as the license is valid. That's why you may need to defend it in court. (Eve may want the court to decide that the license never gave the right in the first place.)

</comment></comment></comment></comment></comment></comment></comment></comment><comment><USER9> I provide support for my code under a "pay it forward" license. i.e. if I provide support to you and help you get my code running, then I expect you in turn to invest the time to clearly document what I showed you. This is a principle of least effort, insofar as I will provide support assuming I won't need to duplicate my work for the next guy. It also means I don't need to needlessly document things that people don't care about. This is similar to lazy coding: "throw an exception if this tricky corner-case pops up, and only if this exception trips will I go through the effort to think through correct behavior for this corner case".

[http://thisscientificmethod.com/2010/02/28/this-is-an- experi...](http://thisscientificmethod.com/2010/02/28/this-is-an-experiment- in-scientific-methodology/#comment-37408278)

<comment><USER10> I like this idea, but why do you link to a blog comment of yours that says exactly the same thing?

</comment></comment><comment><USER11> Is there any legal difference between using this license and releasing something under public domain?

<comment><USER12> In some countries there isn't a legal concept of public domain. I don't know much more than that but I think Germany may be one of them.

<comment><USER13> Yes and no.

Here in Germany (and other countries) we _do_ have a public domain ("gemeinfrei", "gemeinfreie Werke"), but it contains only works whose authors died at least 70 years ago. It is impossible for a living person to actively put their work into the public domain.

However, this is not as bureaucratic as it may sound. You can still sign off any exploitation rights ("Verwertungsrechte")! But there are other rights you cannot sign off, the _author's moral rights_ ("Urheberpersönlichkeitsrechte").

In general, there are two flavours of copyright law in the world:

1) The _copyright_ which centers around the act of copying or doing other things with the creative work. This is found in the USA and Great Britain.

2) The _author's rights_ ("Urheberrecht", "droit d'auteur") which centers around the creators. This is found in Germany and France.

</comment></comment></comment><comment><USER14> I released some code under a similar license, but I just made it up on the spot: <http://trailbehind.com/comment_widget/>

"CommentWidget isn't released under a particular open source license. You can do whatever you like with the code, anything at all."

<comment><USER6> Isn't this a particular license in itself?

</comment></comment><comment><USER15> These kinds of licenses are cute, but you shouldn't use one as it just makes life harder on those that might adopt and patch back into your software. Additionally, in this case, the lack of a disclaimer of warranty in this particular cute license is ill advised.

<comment><USER16> <http://sam.zoy.org/wtfpl/> Why is there no "no warranty" clause?

The WTFPL is an all-purpose license and does not cover only computer programs; it can be used for artwork, documentation and so on. As such, it only covers copying, distribution and modification. If you want to add a no warranty clause for a program, you may use the following wording in your source code:

      /* This program is free software. It comes without any warranty, to
     * the extent permitted by applicable law. You can redistribute it
     * and/or modify it under the terms of the Do What The Fuck You Want
     * To Public License, Version 2, as published by Sam Hocevar. See
     * http://sam.zoy.org/wtfpl/COPYING for more details. */

How does this make life harder for hackers? Explicit permission is given to do whatever the fuck [you] want to, which includes hacking.

<comment><USER15> Watch: I choose to ignore the warranty clause per clause 0. Also, since that clause exists, I can choose to apply it to other code I combine it with (unlikely, but its a dumb license that way)

Past that, I'm not convinced that due to clause 0 that it can be combined with other code, and I am of the opinion that more licenses are not good for open source development in general.

A responsible coder, given the desire in the wtfpl, would adopt a richard hipp style posture, or choose mit,bsd or apache.

</comment></comment></comment><comment><USER17> I always thought this license was misnamed since it does have one restriction:

      "and changing it is allowed as long as the name is changed."

<comment><USER18> _Everyone is permitted to copy and distribute verbatim or modified copies of this license document, and changing it is allowed as long as the name is changed._

Doesn't that just apply to the license document -- not the software itself?

<comment><USER17> Oh, you're right.

</comment></comment></comment><comment><USER19> I would worry that this is not actually legally binding.

<comment><USER20> Why would it not be?

<comment><USER19> If your license won't hold up in court, then you might as well not bother with one. You really can't be cavalier about the langauge in your contracts.

Take the recent landmark case of Jacobsen v. Katzer -- Jacobsen nearly lost because he was using a slightly oddball open source license (the Artistic License) that was not written by a lawyer, had some ambiguous passages, and didn't quite say what he intended.

Sure, it's less of an issue with an extremely permissive license like this one, but let's imagine you aren't around any more and some company wants to use your code... but they can't because their legal department doesn't trust the license.</comment></comment></comment>


Show HN: JD3Sankey to build Sankey diagrams using a REPL interface - <AUTHOR> https://github.com/prathik/jd3sankey/blob/master/README.md <comment><AUTHOR> Please drop any feedback regarding this.

Few questions:

1\. Do you find this useful? 2\. Is the usage simple? 3\. Would you use this for tracking your income and expenses?</comment>


Early 'unicorn' employees can't always cash in - <AUTHOR> http://fortune.com/2014/08/19/early-unicorn-employees-cant-always-cash-in

<comment><USER2> Problem: employee can exercise the options, but must do so immediately and must pay tax on them when they are exercised. Employee can NOT sell the shares now.

Solution: Provide the employee with a loan sufficient to cover the tax burden (to reduce risk, make it no MORE than enough to cover the tax burden), which is guaranteed by a lien on the shares. Employee still owns the shares, and the company making the deal must shoulder several risks including the risk of the stock price falling, but can charge enough to make up for these risks.

With such companies there is often a large interest in purchasing shares which the market cannot satisfy (because the shares are not available publicly), so perhaps you structure the deal so that the loan gets paid off in SHARES instead of dollars. Then the lender could finance it from people with an interest in the shares.

Would this work? If not, why not?</comment>


ReadOps: A virtual book club for people interested in Ops - <AUTHOR> http://readops.com/

<comment><USER2> Ops can mean a lot of things...

IT operations? Business operations? Military operations?

There's zero content on the site you submitted. I'm guessing it's yours, because I can't imagine why anyone would submit a nearly blank site.

Below is a link to the Google Group, also zero content unless you join. Why would I join a random Google Group that I have no idea what it displays or who else has joined it?

[https://groups.google.com/forum/#!forum/readops](https://groups.google.com/forum/#!forum/readops)

Edit: Additional content was added and the group was made public. IT ops.

<comment><AUTHOR> Sorry -- we're still setting up, so things are a bit muddled. I've moved the welcome message onto the homepage and made the discussions public, so you should be able to get a better idea of what's going on now. Thanks for the feedback!

<comment><USER3> You could have taken the opportunity to give us some info about the group when you commented.

EDIT - The welcome screen is still useless as I had to "click" another link to get an idea what the group was about...

</comment></comment></comment><comment><USER4> Well, if I understand it correctly the idea is awesome and I wish it existed for other topics as well. Seems like it would be a great addition to a big providers book catalog like kindle or google books, audible etc.

But as others have said, it could really use more details on the home page.

</comment><comment><USER5> Interesting concept, I'd love to see the same thing for developers as long as the book choice is pertinent (not someone just trying to get Amazon credits).

</comment><comment><USER6> tl;dr: Well, nothing: One must join to view. Why I should I join? What is there to view? I'll never know unless I join, catch-22.

(Should I add FU?)</comment>


Poll: How much of a salary cut would/do you take to work remotely? - <AUTHOR>

I currently work remotely for a yearly salary > $100k, but a friend recently told me I could be making twice that if I joined his company in San Francisco.  It got me thinking, but even though I love San Francisco and spend about half my time there anyway, it would take more than a $100k/year salary increase for me to go to an office every day.  The actual number it would take for me to do that would be about an extra $200k/year.
If you currently work remotely, how much would take for you to go back to the office?
If you work in an office, how much of a pay cut would you take to be 100% remote? <comment><USER2> I work remotely and it would require at least a 100% pay-rise for me to work in the office every day

<comment><USER3> Exactly, I am more efficient working remotely __and not dealing with the BS, so I am not sure what it would take, but the result would not be worth it for the company hiring me.

Why do so many operate under the assumption that remote workers are a handicap?

 __I have a separate dedicated office building, and don 't work from my bedroom, living room, etc.</comment></comment>


Facebook improves its AI Messenger assistant 'M' with new wits - <AUTHOR> https://techcrunch.com/2017/06/27/facebook-improves-its-ai-messenger-assistant-m-with-new-wits <comment><USER2> I thought M was mostly human-powered.</comment>


Overcoming Impostor Syndrome - <AUTHOR> http://www.jeanhsu.com/?p=304 <comment><USER2> _> In college, I never took a course on Compilers or Programming Languages...

>I got the job through a summer internship, which allowed me to bypass the rigorous 8-interviews-in-a-day hiring process. And I lucked out... completely escaped having to code over the phone._

I'm not trying to troll. However, I honestly do consider you landing a Google engineering internship/job with those school background (and zero coding experience outside of school up to that point) being pretty lucky. Consider this guy[1]. I believe he would've most likely gotten the internship had he gone through the same interview process as you. I know that's just my subjective opinion. However, the bottom line is he went through a very different (arguably harder) interview process than you did. It's harder in the sense you could have very well failed had you been asked to code over google doc + phone. The key factor here is this process that Google uses everyday to turn down countless competent but not extraordinary people - you did not have to go through it. That is the basis I deem you lucky.

There is a certain level of respect (among other befits) given to former or current Google engineers among hackers. And I would argue that is the case largely because of the very difficult process that they had to through to become one. I can totally understand why you had the sense of inferiority and insecurity of your coding abilities, having not gone through that process.

Of course none of this makes your point of everyone is good at different things and you should not beat yourself up any less significant. I agree with you on most other points. I'm also happy for you that you've overcome these self defeating thoughts and have proven to yourself. As some one else mentioned in the comments, the problem was you being you own toughest critic.

[1]: <http://news.ycombinator.com/item?id=2384018>

<comment><USER3> This of course brings to mind Yegge's interview loop theory: [http://steve- yegge.blogspot.com/2008/03/get-that-job-at-goog...](http://steve- yegge.blogspot.com/2008/03/get-that-job-at-google.html)

<comment><USER4> Argh, and I was contemplating applying for a product manager role at google. After reading this i wont be applying anytime soon -it's been over 10 years since I studied data structures and algorithms.

<comment><USER3> That article is 3 years old. Don't let one article dissuade you! You should apply anyway if you want to work there.

</comment></comment></comment></comment><comment><USER5> What do you call the opposite of impostor syndrome? When someone thinks they made it to their awesome position by being awesome, whereas in fact they were just lucky?

Alternatively, how about the syndrome where someone blames bad luck for their failures, but in fact their failures are really just due to the fact that they personally suck?

<comment><AUTHOR> according to wikipedia, this is called the "dunning-kruger effect": <http://en.wikipedia.org/wiki/Dunning%E2%80%93Kruger_effect>

<comment><USER6> That is a related but different phenomenon. What you're looking for is called "self-serving bias." [https://secure.wikimedia.org/wikipedia/en/wiki/Self- serving_...](https://secure.wikimedia.org/wikipedia/en/wiki/Self-serving_bias) The dunning-kruger effect is much more subtle than that, take another look at the wiki article on it.

Also related, the actor-observer bias, which describes the opposite effect, and has had more research done on it. This is a common problem in psychology, humans are so full of contradiction.

[https://secure.wikimedia.org/wikipedia/en/wiki/Actor%E2%80%9...](https://secure.wikimedia.org/wikipedia/en/wiki/Actor%E2%80%93observer_bias)

<comment><AUTHOR> ah yes that does seem more accurate. thanks!

</comment></comment></comment></comment><comment><USER7> Great article.

Wanted to comment on this "The first project I did on my own was jeansbargains ... I built it and then realized that I did not want to spend the time looking for deals to put on it"

This seems to run as a common theme in many engineer friends (including myself, several times.) When approached with a sales-like problem, such as a opening a store or a SaaS product, or even starting a social network, we dig into the code first and build, but then get stymied by the "business stuff" necessary to get things off the ground. This might mean not e-mailing potential customers, sales leads, blogging, etc. In the meantime, we'd spend time polishing back-end code that no one will end up using.

I have several unfinished projects due to this syndrome (does it have a name?) And while I am not better at this yet, reading HN has definitely inspired me to do more than just code in that regard.

<comment><USER8> > does it have a name?

The division of labour. Different people have different strengths and preferences. Gains are made from trading on those differences.

<comment><USER9> Also The law of comparative advantage.

<http://en.wikipedia.org/wiki/Comparative_advantage>

</comment></comment></comment><comment><USER10> I'm my own worst critic. I'm never satisfied. Despite countless praise, I always focus on the critique. What can I do better? I'm far too hard on myself, but it's hard to be anything but that. For me, the problem is I focus on what I don't know rather than what I do know. I see what I know as common knowledge, and figure everyone knows it. I see people doing really cool things, and ignore the stuff I've done in the past. So I worry that maybe all the good things people say are merely because I'm good at hiding my inefficiencies.

It's easy to realize you have these problems. The problem though, is it an impostor syndrome, or are you really an impostor? Considering you are always seeing your work as less than perfect, it's easy to believe the latter.

I see this in so much that I do. I cannot just enjoy learning something new. I have to learn the right way, and that means understanding everything. If I'm learning a new language, my first inclination is to rewrite the Hello World example that's posted, because it's not right. So I skip past the introduction and start looking for implementing things properly.

I accept in others supposed deficiencies that I do not tolerate in myself, and it holds me back.

It is, I believe, my greatest weakness.

</comment><comment><USER11> When I started my doctorate years ago, one of the college dons told the first- years that everybody is afraid we'll be "found out", and that somehow our previous track record was merely a fluke. He said these feelings were normal and that we should just let them go. I haven't worried about it since.

</comment><comment><USER6> There was once a billboard campaign on the tube in london that played on impostor syndrome, while managing at the same to be a double entendre.

The slogan went: "Faking it? Try a new position."

</comment><comment><USER12> Too bad I can't admit to feeling a bit of relief upon learning about the impostor syndrome, since doing so would certainly out me as the fraud I am.

<comment><USER13> Hah I had the exact same feeling of relief upon learning that this was an actual thing. But yah, don't tell anyone that.

</comment></comment><comment><USER14> >For others trying to overcome impostor syndrome, consider this: that you have benefited from multiple strokes of luck that have no basis on your actual ability or performance is highly unlikely. It's much more likely that you've gotten to where you are because of your achievements.

uhh, dammit. not helping.</comment>


Ask HN: Are you using MongoDB? If yes, are you a paying customer or open source? - <AUTHOR> If not using MongoDB, what are you using? <comment><USER2> No.

</comment><comment><USER3> Opensource</comment>


Google to simplify messaging strategy, will support only five messaging app - <AUTHOR> https://arstechnica.com/gadgets/2018/12/google-promises-to-shut-down-two-of-its-seven-messaging-apps <comment><AUTHOR> Could someone in Google let us know why they feel the need to kill/mutilate/destroy Hangouts?</comment>


TimescaleDB – an open-source database to make SQL scalable for time-series data - <AUTHOR> https://github.com/timescaledb/timescaledb <comment><USER2> Thanks for sharing. Can I use timescaledb with tableau?

<comment><USER3> Yes you can since it's a Postgres extension; any frontend tool that interfaces with Postgres, interfaces with TimescaleDB. One issue is you will see our "private" schemas where we store things, but you can just ignore those (or play with them at your own peril :) ).

Source: TimescaleDB developer</comment></comment>


Atmospheric railway - <AUTHOR> https://en.wikipedia.org/wiki/Atmospheric_railway <comment><USER2> At the very end, about a current implementation:

> _The free piston is magnetically coupled to the passenger modules above; > this arrangement allows the power tube to be closed, avoiding leakage. The > transportation unit operates above the power tube on a pair of parallel > steel rails. The company currently has a 1 /6 scale pilot model operating on > an outdoor test guideway... The Corporation claims that a full-scale > implementation would be capable of speeds in excess of 200 mph (322 km/h)._

It sounds like magnetic coupling is the solution to the maintenance issues.

But it also seems like you can only have one train on a track at any given time.

Or, at least a track would have to be divided into sections, each with its own pumps and own piston, which could only support one train at a time... and then the piston would have to be sent backwards to the beginning of the section to be ready for the next train. (And you'd need electromagnets to let go of one piston and grab the one on the next section.)

So I can see why this might not be viable for something like a city's subway system.

But at the end of the day... what advantages would this ever have over electric trains that get their power from a third rail?

<comment><USER3> > But at the end of the day... what advantages would this ever have over > electric trains that get their power from a third rail?

No heavy electric motors and other equipment in the vehicle, for starters

<comment><USER4> From the article:

> Fully loaded vehicles have a ratio of payload to dead-weight of about 1:1, > which is up to three times better than conventional alternatives

Might make sense for a small system.

But electric machines have made huge advances in the last decades. Frequency converters, DC-DC converters, high torque permanent magnet motors etc are now very commonplace.

For example in the workshop, electric tools have supplanted pressurized air ones.

</comment></comment></comment><comment><USER5> [https://en.wikipedia.org/wiki/Dalkey_Atmospheric_Railway](https://en.wikipedia.org/wiki/Dalkey_Atmospheric_Railway)

"The journey to Dalkey could be completed in just over 3 minutes with the speed being limited by the need to brake for curves on the"

That's faster than the train gets there today.

</comment><comment><USER6> Related technology: [https://en.wikipedia.org/wiki/Fireless_locomotive](https://en.wikipedia.org/wiki/Fireless_locomotive)

</comment><comment><USER7> I did not know that a modern version of the system was in use. Spent a few years living along the South Devon route, and Brunel's seals eaten by rats were a definite thing.

<comment><USER8> I'm from Porto Alegre. The aeromovel works fine, but I guess an electric train could fit the same constraints.

AFAIK the aeromovel is a matter of "regional pride" since a local engineer invented/designed them in the 70s.</comment></comment>


Count your website's LOC using one Linux command - <AUTHOR> http://karim-ouda.blogspot.com/2012/07/counting-lines-of-code-for-your-website.html

<comment><USER2> ... or you could just use sloccount.

<http://www.dwheeler.com/sloccount/>

</comment><comment><USER3> ... or cloc: <http://cloc.sourceforge.net/>

</comment><comment><USER4> Any shell script can be made one line with enough semicolons.

</comment><comment><USER5> highly under-used: grep -c -e . -r .

</comment><comment><USER6> methinks the point of this is to do it without installing other tools.</comment>


Project Helium: Modernizing Software in Minutes - <AUTHOR> http://blogs.adobe.com/conversations/2015/12/project-helium-modernizing-software-in-minutes.html <comment><AUTHOR> More info in this paper: "Helium: Lifting High-Performance Stencil Kernels from Stripped x86 Binaries to Halide DSL Code" [0]

[0] [http://groups.csail.mit.edu/commit/papers/2015/mendis- pldi15...](http://groups.csail.mit.edu/commit/papers/2015/mendis- pldi15-helium.pdf)

</comment><comment><USER2> I wonder wether it will be able to optimize itself in the future.</comment>


Significant Research papers behind the growth in Big Data Tools - <AUTHOR> http://miningbigdata.blogspot.com/2012/12/the-research-papers-leading-to-big-data.html

<comment><USER2> No relevant research was done before Google and Amazon? That's news.

<comment><AUTHOR> These papers were key milestones and the technologies described therein benefited from previous research.</comment></comment>


Wireshark 1.2.0 Released - <AUTHOR> http://www.wireshark.org/docs/relnotes/wireshark-1.2.0.html <comment><USER2> Does anyone know enough to pick highlights out of the release note?

I like Wireshark as a tool but I'm hardly an expert.

</comment><comment><USER3> Be warned, doesn't like 64-bit Win7; tacket capture driver doesn't work. Nevertheless, looks sufficiently awesome that I'm going to dust off an older machine just to play with it.

<comment><USER4> on 32-bit win7 it was "did not automatically install packet capture driver". I downloaded winpcap 4.1 beta5 seperately, set the compatibility mode to WinXP SP3 and it installed. Doesn't install cleanly for RAS/VPN network connections but wired and wireless adapters work.

</comment></comment><comment><USER5> Experienced few crashes on Windows XP and also GUI, at times acts funny, clicking anywhere on menu, brings the File open dialog box.</comment>


Migrate from Twitter to Mastodon - <AUTHOR> https://rmpr.xyz/Migrating-from-Twitter-to-Mastodon/ <comment><USER2> Bullshit title.

You cannot "migrate" from Twitter to Mastodon in the universally understood meaning of that word in a software context. Because there is no way to take your data with you.

"How to open a Mastodon account" would have been a more accurate title. But obviously not as baiting of clicks!

</comment><comment><USER3> The appeal of Twitter is the people you follow - if they're not in the Fediverse (and believe me, I have tried), you're unfortunately stuck with either staying on Twitter or abandoning those people. Maybe as time goes on people will slowly migrate but I can't see it happening in sufficient numbers unless something catastrophic happens to Twitter.</comment>


Show HN: Checkin – One-question surveys through Slack - <AUTHOR> http://checkin.chat/ <comment><AUTHOR> Hey guys, at the company I currently work (startup 150+ employees) where at a stage in which aligning different aspects of our (newly built) processes and strategies is crucial.

Asking questions to whole team (or a group in particular) is s need that happens on a daily basis and is most often met by a mass email that gets sent around. These often usually come with a Survey Monkey/Type form/Google Form/etc link in it.

I've built this because I've found that asking (in private message) through Slack leads to considerably higher response rates (+50%) and for the responder, there's no new UI to learn every time a new question comes in.

Hope you like it. Looking forward to get feedback!</comment>


What Explains U.S. Mass Shootings? International Comparisons Suggest an Answer - <AUTHOR> https://www.nytimes.com/2017/11/07/world/americas/mass-shootings-us-international.html <comment><USER2> Am I right that the second graph on this page does not support this headline at all? Remove two data points and the whole idea falls apart.</comment>


Why San Francisco Is Not New York - <AUTHOR> http://bits.blogs.nytimes.com/2014/03/20/why-san-francisco-isnt-the-new-new-york/ <comment><USER2> Not to mention, New York is relatively clean compared to San Francisco. San Francisco shuts down at 10pm; New York truly is the city that never sleeps, and many neighborhoods feel busy and alive at all hours of the night. The public transportation in SF (and the larger Bay Area) is a classist, unreliable nightmare. New York has probably the best public transit systems in the country. Owning a car in NY is unusual and unnecessary. In San Francisco, most people own a car, just like any other suburban area in the US. The number and quality of restaurants in NY is unrivaled. New York is unquestionably more diverse. The list goes on and on.

<comment><USER3> > The number and quality of restaurants in NY is unrivaled.

NYC native and mostly SF-hater here. Hate to say it but SF has us beat for food. If you want to go have a 5 star meal you're going to pay about the same and get about the same food in both cities (NYC obviously has the edge here, but not hugely), but San Francisco has NYC (especially Manhattan) absolutely dominated at the everyday eating level (ie: burritos, sandwiches, cheaper restaurants). The general ingredient quality just blows NYC away - both meat and produce.

<comment><USER4> " The general ingredient quality just blows NYC away - both meat and produce."

Well, west coast (meaning: access to Australia and some other selling markets), good climate, etc

<comment><USER5> SF's neighbor to-the-east, The San Joaquin Valley, is "the nation's salad bowl." Which also happens to be home to many cattle and sheep operations.

Access for foreign imports are important for the gourmet scene, but not necessary.

<comment><USER6> "Virtually all non-tropical crops are grown in the Central Valley, which is the primary source for a number of food products throughout the United States, including tomatoes, almonds, grapes, cotton, apricots, and asparagus." [https://en.wikipedia.org/wiki/Central_Valley_(California)](https://en.wikipedia.org/wiki/Central_Valley_\(California\))

Minor correction: The San Joaquin Valley in the south half of the Central Valley, and The Sacramento Valley is the north half. The dividing line is the Sacramento–San Joaquin River Delta

</comment></comment></comment></comment></comment><comment><USER7> I lived in NYC at one point, and I'd agree that the two cities are fundamentally different, but I think he missed what may be the biggest source of the difference: the proportion of people playing zero-sum games. A lot of the most ambitious people in NYC work in finance, whereas in SF hardly any do. Nearly all the most ambitious people in SF make money by creating things that other people are willing to pay them for. That means SF attracts different people than NYC, and changes people who live there in different ways.

You get the money later when you do that, which is another big reason people in SF don't spend so much. In finance you get paid as you go, and a lot of people in NYC spend it as they go. Indeed, friends who've worked in the finance industry have told me firms have a deliberate policy of encouraging new recruits to spend all their salaries and bonuses, as way of getting them hooked. Whereas in SF a lot of people's net worth is tied up in illiquid stock.

<comment><USER8> In don't think "zero sum game" means what you think it means. Lots of people in San Francisco engage in transactions, and lots of people in New York engage in reducing transaction costs. Neither are "zero sum" endeavors. Insurance is a great example. My office used to sit in the shadow of the Met-Life building. They don't make anything, so they must be zero sum. Wrong. Insurance enables transactions to happen that wouldn't otherwise happen, adding value to the economy. That's the crux of many of New York's economic sectors: finance, advertising, insurance, law, consulting. Its abstract, and service oriented rather than product oriented, but its not zero sum.

Now one can argue about the amount of value created by these industries, or rather how much of the value created is absorbed by the industries themselves. You can also question how much value is created by yet another web app getting funded in SV, so I don't know if you're on firm ground there.

<comment><USER9> Having worked in finance, I can promise you that a lot of people in finance are playing zero-sum games. All trading is basically zero sum. No trader goes home saying, "Hey, honey, I imperceptably increased global liquidity today; yay me!" They say things like, "I'm fucking awesome. I gutted those chumps in the LIBOR pit. Whores for everybody!" [1]

Anybody working for those people is also essentially playing a zero-sum game, in that their activity is just aiding in shifting value from one pocket to another. Which is why I no longer work in finance; I wanted to do something useful.

[1] Yes, I have actually heard traders say "Whores for everybody!" in the office. At least in the office they were joking.

<comment><USER10> Even if some traders think that they are playing zero-sum game - it does not mean that traders actually play zero-sum game.

Benefit of trading is increased liquidity.

<comment><USER9> It matters entirely for the psychology of the industry's culture.

However, increased liquidity is of no direct value. It only matters if people get better prices, or get deals closed significantly sooner. For a trader's participation to be positive sum, the benefit of their capital has to be larger than the costs they impose. Given that every trader is working to make the opposite true, surely many of them are negative sum. Especially given that capital can be deployed to reduce liquidity just as well as increase it.

</comment></comment></comment></comment></comment><comment><USER11> The other difference? Layout.

If New York were the Bay Area, then all the big tech employers would be in places like White Plains and Greenwich, while all the tech employees still would want to live in Chelsea and the Village and the usual spots. Theoretically there's public transit from one to the other, but it's really a bit of a strain when you get down to it, so they'd need to run private buses.

New Jersey would replace Oakland and the East Bay - downmarket reputation but you've got a few excellent clusters of local culture, and you can get into town from there easily enough for your commute. You just don't actually _feel_ like integrated with the rest of town where everything is happening. Staten Island replaces Marin, eastern Long Island replaces wine country...

and the entirety of Brooklyn and Queens would be underwater. :P

</comment><comment><USER12> Year after year, decade after decade I hear "SF is the next this, that or the other". I believe SF is the city with an inferiority complex. No matter how accomplished SF becomes, it can't simply be itself, it has to take somewhere else down. I've given up the discussion at social events; SF simply want to be anywhere else but itself, and that creates all kinds of f'ed up logic spewing out of it's citizen's mouths.

<comment><USER13> One of the strongest examples of SF's inferiority complex is how some residents insist on capitalizing "The City" mid-sentence when referring to it.

</comment></comment><comment><USER14> > So what exactly is San Francisco? When I came across a passage in the book, > "The Annals of San Francisco," about the 1840s Gold Rush, I found the answer > to that question.

So SF is bunch of gold diggers?

</comment><comment><USER14> > The lack of diversity between social groups in San Francisco isn't going to > change anytime soon, as the number of tech employees in the Bay Area is only > going to continue to rise.

I am not sure I understand. Tech sector brings people from all over the world including India, China, South America and Europe.

<comment><USER15> Real diversity is not how you look or where you come from, but what's inside you and how you think and the culture you belong to.

<comment><USER14> It kind of sounds like 'No true Scotsman'. Could you explain what sort of diversity you miss in tech sector?

<comment><USER16> I think the argument is that if everyone in SF is tech focused now, despite different backgrounds, there is a lack of diversity in the interests of the groups taking over SF (techies), where there used to be diversity. So now, everyone's a techie, instead of maybe musicians or artists, etc. I'm not sure it's a good argument, but I think I agreed with it briefly.

</comment></comment></comment></comment><comment><USER17> I wouldn't mind an article titled: "Why New York is Not San Francisco"

<comment><USER18> One isn't needed. New York is very happy being New York. It's San Fran that isn't happy being San Fran.

<comment><USER19> Having lived in SF since 1996, I'd like to formally refute your point here. I'm quite happy with SF. I also live outside the Mission gentrification world, but do work in the tech industry. SF is still weird, it's still got its distinct character and it's still part of an incredible geography. I like being in the West because it's as much a psychological distance as anything. I say this as someone born and raised in Virginia.

I'm always amused at people's attitudes about San Francisco, and San Franciscans. Keep 'em coming.

</comment></comment></comment><comment><USER20> It's easy to confuse the negative effects of scale with cultural similarities. I've lived in both cities and you'd have to step outside the US to find two cities less like each other.

Its easy to understand why New York magazine would publish high-brow link- bait: the decreasing relevance of banking-- NY's primary economic engine.

The funny thing is that the importance of banking is cyclical. Banker misbehave, markets punish them, they pretend to cower, they quietly recover... and then they misbehave again.

These kinds of comparisons will diminish once it's misbehaving time again.

<comment><USER21> The tech:SF::banking:NYC comparison is false for a number of reasons. Chiefly, no single industry in New York overwhelms the city the way tech overwhelms San Francisco. Finance, fashion, media, and many other industries are very big in NY. But NY is still bigger, bigger than people from San Francisco can fathom.

San Francisco is already overwhelmed with tech culture, and this sector continues to grow. Everywhere you turn, it feels like you meet people who work for startups or large software firms. There is no comparable industry in NY that is so pervasive. This lack of economic diversification breeds the other diversity deficits this article mentions.

<comment><USER22> Well said - NYC is big in many different facets. It has a large tech presence, but when considering all the other large presence of industries, it's just another sector.

To give an idea, companies like IBM and Pepsico (Pepsi, Frito Lays, etc.), and Kraft have a large presence just north of the city and/or in it. NYC is one of the largest cities for fashion. It is home to the finance capital of the world. It has one of the largest music scenes in the world. It has a huge advertising agency presence.

There is a lot more than just the aspects I mentioned even. That is how massive of scale NYC is - San Francisco's legacy is primarily tech.

</comment></comment></comment><comment><USER23> ITT: Subjective 'Facts' and Emotional objectivism.

</comment><comment><USER24> It hardly seems likely that NYTimes could be unbiased here.

</comment><comment><USER25> The only reason I would pick SF is because of the weather.

<comment><USER12> you like freezing? you've confused SF with southern california; worlds apart in attitude and temperature: laid back and warm.

<comment><USER26> Having lived through this winter in the Northeast, I can promise you that SF is beautifully warm all year round. Case in point: it is a place where people use the word "freezing" to describe temperatures 10-20 degrees above the melting point of water.

</comment></comment></comment><comment><USER27> ok?</comment>


My journey applying AI to horse racing - <AUTHOR> https://towardsdatascience.com/applying-ai-to-horse-racing-e3632a7e7c92 <comment><USER2> This is another case where being rich lets you get richer. :)

I had a friend who had a private box at a horse track. One of the benefits of this box was that you had a personal betting machine. This let you bet on the best odds just seconds before the race every time. I made a lot of money doing that.

You can kind of do it in Vegas too because the sports books aren't usually that busy.

In fact, that hardest part of this strategy is actually figuring out who the favorite is! The odds are usually listed as fractions, but they have different denominators, so you quickly have to convert them all the the LCD and then order them.

But yes, the closer you bet to post time, the better your outcome with this strategy will be.

<comment><USER3> The closer to the off the more accurate the price is (wisdom of the crowd) thus less value, you need to know something others don't to win long term from taking the starting price!

I assume you are American as everyone in the UK has a personal betting machine and it's in their pocket.

<comment><USER4> Not sure how it works elsewhere but horse betting in the US is almost always parimutuel, meaning that everybody gets the post odds, regardless of when they bought their ticket. In other words, betting early doesn't give any advantage and in fact the later you can wait, the more accurate an idea you have of the actual payout odds of your bet. Thus, having access to a private betting machine, and not having to elbow other gamblers to buy a ticket at the last minute, is a real advantage.

That said, the state's take-out is so high in most states that you need a massive edge to actually make money. The slight edge a private betting machine would bring probably isn't enough.

<comment><USER5> If you bet at 10-1 here, your bet is 10-1 on your receipt, and your odds are 10-1. Your bet at that price may change the odds for everyone who comes later, but that doesn't affect yours. It is quite possible to arbitrage odds between bookmakers. Or it would be, if they didn't do such a good job of conferring and mutually underwriting their risk themselves.

It must be crazy having the odds change on you after your bet.

<comment><AUTHOR> For Swedish harness racing there is only one bookie. The state has had monopoly on betting in general for a long time and is very strictly regulated also today. So, there won't be any arbitrage across bookies for these races at least.

<comment><USER6> What if... bear with me... somebody outside of Sweden took bets on Swedish harness racing?

</comment></comment></comment></comment></comment></comment><comment><USER7> Actually the most successful parimutuel syndicates and HNWI bet programmatically through API's. They consistently win and they do not make their money from picking a single winner. They bet on exotic bets that have much higher odds and are easier to box. For instance picking two winners or order of the first three horses.

Source: I worked on those APIs to integrate with 20+ exchanges over three companies and 10 years. The syndicates and HNWIs were always the same people over that decade.

<comment><USER8> That sounds fascinating. I don't suppose you have a blog, or any other write up about that world? Even perhaps a link to one of these betting APIs, if they're generally available to the public?

<comment><USER7> Betfair has a public API, but they are not parimutuel: [https://docs.developer.betfair.com/display/1smk3cen4v3lu3yom...](https://docs.developer.betfair.com/display/1smk3cen4v3lu3yomq5qye0ni/API+Overview)

Tabcorp is an Australian Wagering Company that offers real-time horse racing APIs: [https://studio.tab.com.au/](https://studio.tab.com.au/)

Others are a little less public and you need to sign up with an account and prove you are a syndicate. Hong Kong Jockey Club, UniBet, Amtote.

I don't normally talk about this history as I was active in this industry until four years ago. There are however people who have come out in public:

Bill Benter: [https://www.bloomberg.com/news/features/2018-05-03/the- gambl...](https://www.bloomberg.com/news/features/2018-05-03/the-gambler-who- cracked-the-horse-racing-code)

Jelko: [https://www.smh.com.au/national/meet-the-joker-the- australia...](https://www.smh.com.au/national/meet-the-joker-the-australian- who-is-the-biggest-gambler-in-the-world-20180515-p4zfhi.html)

Most of the other families and players are extremely private or are of Eastern European I wont talk about them publicly.

This is a list of common bet types that are targets for these types of players: \- Trifecta \- Big Six \- Quadrella \- First 4

Here is a blog on working the combinations of the Big Six: [https://practicalpunting.com.au/pp-online/a-z-of- betting/exo...](https://practicalpunting.com.au/pp-online/a-z-of- betting/exotic-betting/big-six/my-formula-to-beat-the-big-6-20090404)

<comment><USER9> hah, how funny I almost copied your post word for word ;)

What are you up to now Shaun? Are you still in Hobart?

</comment></comment></comment></comment><comment><USER10> He has this completely wrong. The idea in pare-mutuel betting is not to pick the winner. The favorite usually wins, but the house cut makes betting the favorite a net lose.

The goal is to find underpriced bets, where the collective estimate of the odds is far from the prediction systems's estimate.

<comment><USER11> In this case there is no house. For this to work a player needs to be able to back or lay an outcome.

"In harness racing, you are playing against all the other players. You are not playing the house. The odds of any given horse winning is directly related to the amount of money on it."

<comment><USER12> I'm pretty sure these races are pari-mutuel where the house takes a vigorish. The sentence you give is basically the definition of pari-mutuel without the vig.

FWIIW the author is wrong that this problem is unstudied. There are actual hedge funds which do nothing but this and a quick google search provides numerous academic examples; even a github repo. "Deep learning" is probably the worst approach I can think of to the problem.

I'm also guessing he misunderstood the data if betting on the favorite compounds like that. If you make the race pay off at something other than the actual odds, that would probably do it.

Examples from the literature:

[https://github.com/dominicplouffe/HorseRacingPrediction](https://github.com/dominicplouffe/HorseRacingPrediction)

[https://www.sciencedirect.com/science/article/pii/S016792361...](https://www.sciencedirect.com/science/article/pii/S016792361200379X)

<comment><AUTHOR> It doesn't say that horse racing is understudied. It is saying that harness racing is understudied. Normal horse racing is very different to harness racing.

Also. The simulated betting on on the favourite on those 26000 races was with one dollar per race. If the favourite win, the balance increases with 1$*the odds. If the favourite does not win, the balance decreases with 1$. Nothing strange about it, and as I tried to explained, no compounding involved. Flat betting. 1 $ per race. Anyway. It was simulated on historic data and also a side note as it didn't really have anything to do with the project itself. Appreciate the comment.

<comment><USER13> I don't mean to sound rude, but I don't believe the results in your $1 * 26000 races graph.

I spent a few years in a hedge fund building algorithmic trading strategies, and whenever I saw a graph (we called them 'account curves') like that, it inevitably meant that there was a bug somewhere. Usually the strategy is somehow non-tradable, either because the market is closed or the system is borrowing data from the future.

The graph shows you making about $55,000 over 26,000 races (i.e. about a 100% return) with no big swings. If the effect was that big, wouldn't people know about it already? Despite the fact that you can't apply the strategy _everywhere_ , couldn't you settle for making a little money at your local betting shop?

Your theory is that there are more informed bettors betting at the last minute. That suggests that the market odds spike towards the 'correct' odds as betting is about to close. If someone proposed your 'bet on the favourite' strategy to me, I would have asked to see evidence of that spike. Are you able to get access to real-time odds to test the theory?

<comment><USER14> Betting $1 on 26,000 races:

Stated win rate betting on the post time favorite is 37%

This equates to a loss of 16,380 of original 26,000. Earnings must then come from the 9,620 bet on favorites that won.

Assuming the 55,000 is winnings (and not a total including original 9,620), this would equate to an average odds of better than 11/2\. If reduced by the 9,620 then odds come down to a bit over 9/2

My experience in horse racing (regular and harness) is that odds on the favorite of 11/2 (or even 9/2) are extremely rare and certainly not the average. A realistic figure is 3/2 over the various field sizes (smaller fields tend to go close to even money, larger to 2/1 or so).

So yeah, the data as presented appear flawed.

</comment></comment></comment></comment></comment></comment><comment><USER15> Using computers to beat masses of the general public at horseracing in this fashion has been going on for well over 30 years. I first read articles about the gambling teams and data scientists behind them in Hong Kong in the mid 90's. The teams had data capture people entering realtime odds from the tracks at Sha Tin and Happy Valley and hired private tellers to enter thousands of computed bets moments before a race began from their offtrack locations in downtown HK highrises.

The thing that makes it easy to simulate but impractical to execute is that getting enough bets in fast enough and late enough to tip the odds to your favor to give you an edge while limiting losses is difficult to do. It's even harder to do without this wagering itself affecting the odds. For one there often just isn't enough money wagered on a race. In the heyday of the HK races you'd often have 100 million+ HKD wagered on every single race, much of that coming from the general population -- individuals going to the track. There are only a few big event races in the US or Europe that will attract that kind of betting pool. Even HK is no longer like that.

<comment><USER16> Bloomberg had an article on this, last year.

[https://www.bloomberg.com/news/features/2018-05-03/the- gambl...](https://www.bloomberg.com/news/features/2018-05-03/the-gambler-who- cracked-the-horse-racing-code)

</comment></comment><comment><USER17> > The last results, after remodeling the problem, came to be on par with the > odds themselves. Around 37%. This time my AI was as good as consensus at > betting harness racing.

using the same dataset as betters with a NN is not really better than running a regression on it, so this doesn't really surprise me.

nn are able to find patterns with things others than historical series. taking this forward would need to combine historical race sets external sources that could have correlated information available that's not easy to extract by simple regression, like news and articles whenever available or meteorological data or whatever, just feed it everything and let the nn sort itself the useless data out

<comment><AUTHOR> That's in fact what was done. Facts about the races (everything except for the actual odds for the horses) combined with Google Maps API. It basically predicted the odds of the horses (more or less).

</comment></comment><comment><USER18> I found this nugget to be the most valuable:

 _Simulated flat betting 1$ on 26 000 races over 15 years shows a hefty return on investment_

If this is true, why don't we all just go and hire people in Sweden to put in bets for us at the last minute, based on the favorite horse, and pay them out of the profits? We pay them to do it and we collect the winnings.

If this strategy works it may work elsewhere too. Why not just do that all day long with an army of people? And you aren't hurting who is the favorite either.

<comment><USER19> well, for once, just because it works it doesn't mean it's worth it. a couple of years ago i read a story about a man who cracked a scratch-off lottery ticket code (the serial numbers were biased). he could have earned more than with his day job if he devoted his time fully to gaming the system, but ... it decided it wasn't good enough - a mindless, robotic task, ending in misery soon.

the take away of the story was that the method was probably known by crime syndicates and used for money laundering, as they have cheap pseudo-slave labour available. but if you're a well educated person with a fulfulling job, you'd go insane soon (and of course, job security - the second the company changes their system you're out).

so with betting, you'd have to invest a lot of time and energy for it to become profitable. as he wrote, you'd have to be at the race tracks and wait until the last possible moment, then driving to the next stadium. and if it was _that_ easy to create a real winning system that depended on local help, the swedes would do it themselves - so your hired goons might stay as long as they needed to understand your system and then continue on their own (lowering your margins in the process).

if the process could be completely automated, i.e. with online betting, the story might be different. but i'm sure there are already a lot of syndicates looking into that (even the analog version). for them it doesn't even have to be profitable. 80% return is good enough for money laundering and they can out-finance the legal competition.

<comment><USER20> I remember a very similar story, it stuck with me from a couple of years ago:

> His next thought was utterly predictable: "I remember thinking, I'm gonna be > rich! I'm gonna plunder the lottery!" he says. However, these grandiose > dreams soon gave way to more practical concerns. "Once I worked out how much > money I could make if this was my full-time job, I got a lot less excited," > Srivastava says. "I'd have to travel from store to store and spend 45 > seconds cracking each card. I estimated that I could expect to make about > $600 a day. That's not bad. But to be honest, I make more as a consultant, > and I find consulting to be a lot more interesting than scratch lottery > tickets."[1]

[1]: [https://www.wired.com/2011/01/ff- lottery/](https://www.wired.com/2011/01/ff-lottery/)

</comment></comment></comment><comment><USER21> Interesting comparison to the old fashioned way: [https://www.bloomberg.com/news/features/2018-05-03/the- gambl...](https://www.bloomberg.com/news/features/2018-05-03/the-gambler-who- cracked-the-horse-racing-code)

</comment><comment><USER22> The OP writes repeatedly that the "odds of winning are related to the amount bet on each horse." This is not correct. The _payout_ is related to the number of people betting. The odds of winning are based on the horse's DNA, training, competitors, etc. The horse does not know who bet it on it while it's running. It's easy to confuse the word "odds" in gambling because that often refers to payout, but these are not the odds of winning.

</comment><comment><USER9> Here's my 2 cents.

Anyone who is telling you they have cracked the parimutuel pools, and "here's how to do it", is trying to sell you something else. Anyone who has actually cracked the parimutuel pools knows how they work, and that if you share your "tips", or selections, you only serve to diminish your own returns, so you are not going to do it.

<comment><USER22> Indeed. This is the origin of the expression "break a leg." At the track, you could never wish someone else good luck because their good luck will cost you money. Instead, you wish their horse breaks a leg so he'll be eliminated, moving the money in his pool to yours if you win.

</comment></comment><comment><USER23> He was unlucky in picking harness racing. Standard horse racing is already corrupt, but harness racing takes it to the next level. The rules and the addition of the harness make cheating much more prevalent.

I'd rather bet on Professional Wrastling than harness racing.

</comment><comment><USER24> appreciate the write up but is this even AI? It seems like he just did some analysis & stats on observed data, and came up with a simple model.

<comment><AUTHOR> The article doesn't really say, as this was a write up of a 20min talk at a conference for economists. The main model where vanilla ANNs. Also SVMs and LSTMs where tested but unsuccessfully. As I had a lot prior experience writing Java, most models where implemented in the beginning in DeepLearning4J but was in the later stages moved over to the frameworks from Fast.ai. I'd be happy to do a write up of the details at some point. The majority of work was in fact the data pre-processing.

<comment><USER24> Ah, no need to give me all the details good sir. Appreciate the post. Also yeah, sounds about right with the pre-processing. That is always the case!</comment></comment></comment>


Sergey Brin: The Future of AI and Google - <AUTHOR> https://www.youtube.com/watch?v=XzkUAxtEQXE <comment><USER2> Thanks for sharing this. I'm interested to hear what Monsieur Brin considers relevant in such discussion.</comment>


Ask HN: What's your churn rate? - <AUTHOR>

A question to web-application providers: what percentage of your clients quit your service every month? is the rate different for free accounts compared to paying accounts?<p>p.s. Any links to some stats? <comment><USER2> Here are my numbers for S3stat. We don't have free accounts, though we do have a 30 day trial.

      Visit to trial conversion:   3%
      Trial to paid conversion:    26%
      Average subscription length: 11.5 months
      Average cancels per month:   2%

Over time, those last two numbers keep getting better, since there's more time passed for our first loyal customers to inflate the average subscription length, and there's more people on board to lessen the impact of a few people cancelling.

It's worth noting that you only need to calculate one of those last two numbers to know the expected lifetime value of a visit to your site. Until you asked the question, I had never actually calculated my churn rate as a percentage of active users.

<comment><USER3> Interesting stats. How long have you been running S3stat?

<comment><USER2> The beta went live at the end of 2007, and I started charging for it in February 2008.

S3stat goes squarely against one of the biggest pieces of advice I give to fellow startup kids: Never base an entire product off a missing feature in somebody else's thing.

Thus far Amazon have been good in keeping their AWS reports in the same state of disarray I found them in 3 years ago. I don't expect it will stay that way forever.

<comment><USER3> I'm sure by then you will find other successful niches :)

</comment></comment></comment></comment><comment><USER4> No public stats I can personally provide. But I have seen it vary wildly depending on the type of service. Businesses are generally slower to adopt a new service/tool, and once they are locked in can be apprehensive for change. Consumers can change on a whim if something better comes along.

Here are some interesting stats I found for Evernote: [http://techcrunch.com/2010/05/28/video-evernote-ceo-phil- lib...](http://techcrunch.com/2010/05/28/video-evernote-ceo-phil-libin-shares- revenue-stats-and-how-to-make-freemium-work/)

Evernote has been pretty open in their stats over the course of their existence.

<comment><USER5> It can also vary a lot within a type of service depending on the target market. At Braintree our churn rate is higher for the start-ups that we work with (because they sometimes go out of business) than it is for companies that have been around longer and are profitable.

</comment></comment><comment><USER3> Constant Contact is a public company, there for they share stats in their annual report [http://investor.constantcontact.com/common/download/download...](http://investor.constantcontact.com/common/download/download.cfm?companyid=CTCT&fileid=368991&filekey=8D782BD5-08BC-4DF8-9DA2-A5698A49E40B&filename=CTCT_09AR_4.26.10.pdf)

Some key stats. Their churn is ~2.5% and their cost of acquiring customer is (whopping!) $370

PS: planning to write a post analyzing all such open stats. Do you know any other public Internet companies?

</comment><comment><USER6> This might help too: for really crap services (say, weightloss or the like), where people sign up and then pretty much never use the service, a typical average churn rate is 3 months. ie. it takes people 3 months on average to unsubscribe from a service they don't use.

<comment><USER3> Are you speaking out of experience or do you have some stats? It seems unlikely that people take at least 3 months to unsubscribe a service they never use.

<comment><USER6> Out of experience with 2 large services. Can't share stats, sorry. It may seem unlikely, but it's true :) I'm talking average, many take up to 6 months. Now you know why there are so many ads for crappy services online :) ps: I didn't say "at least", I said "on average".

<comment><USER6> The math of subscription services generally goes like this: acquiring a paying customer is going to cost you about 15-60$ each (for typical lifestyle services). So lifetime value needs to be more than that. Once you're live, you tweak those numbers, make acquisition more efficient and increase lifetime value.

</comment></comment></comment></comment><comment><USER7> Is there an equivalent or similar metric for non-subscription commerce sites? I operate one, and I'm struggling to find good references on the best way to determine I've 'lost' a customer.

<comment><USER8> Im working on a service, currently in private beta, that I think you (and lots of others here on HN) would be interested in.

Shoot me an email and I'd love to tell you more about it: Pospischil at gmail</comment></comment>


Ask HN: Resources on vitamin D supplements and studies - <AUTHOR> Hi there, would you share your experience with taking vitamin D supplements and resources & studies you found about the subject? The only good summary of research I've found is from several years ago: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2908269/<p>What did you take it for? Did it work or improve your condition? Any regrets ("not taking it sooner" counts)? <comment><USER2> Examine.com is a good starting point [https://examine.com/supplements/vitamin-d/](https://examine.com/supplements/vitamin-d/)

If you scroll down to human effect matrix you can find links to studies based on results found.

The summary at the bottom is also pretty good.

<comment><USER2> To add some anecdotal evidence... We basically get no sun for half of the year where I live. I find my subjective wellbeing has increased somewhat, especially during the winter months, since I started supplementing with 5000UI D3.

Other potential benefits are harder to measure, but the cost in terms of money and likelihood of adverse effects is so low I feel I might as well continue.

To add another data point, gwern has done some more rigorous self experiments [https://www.gwern.net/zeo/Vitamin-D](https://www.gwern.net/zeo/Vitamin-D)

</comment></comment><comment><USER3> There was a bit of research just recently that I believe was posted here about the RDA values for vitamin D being far lower than they ought to be due to statistical errors in past studies. The errors resulted in underestimating the levels of vitamin D necessary to avoid negative health effects pretty dramatically, and they recommended changes to health and diet recommendations ASAP. I believe that they estimated that most people should probably be taking 10,000 IU/day supplementally.

Personally I was diagnosed as vitamin D deficient a few years ago and my doctor prescribed gelcaps that were 50,000 IU to be taken twice a week. Upon switching doctors, my new doctor noted the deficiency in my records and said that she recommends everyone in my state (West Virginia) take vitamin D supplements as nearly everyone is deficient. And that was by the older, lower, standard. She didn't renew my prescription but I have begun taking 10,000 IU/day OTC supplements, especially after reading that research I've been considering doubling up.

</comment><comment><USER4> I always shrill about my passion project whenever I see this, but I suffered a lot of issues of quantifying supplements and habits to improvement so I wrote [https://betterself.io](https://betterself.io) (open-sourced)

It proxies a lot of improvement to productivity and sleep, (which are somewhat flawed as metrics).

</comment><comment> open-source-ux _"...resources & studies you found about the subject"_

The UK's Scientific Advisory Committee on Nutrition (a body that advises the UK government on health matters) published a review on vitamin D and health in 2016.

They recommend that everyone over one year of age should consume 10 micrograms of vitamin D daily.

The full report is 304 pages (PDF, 4.2MB)

[https://www.gov.uk/government/uploads/system/uploads/attachm...](https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/537616/SACN_Vitamin_D_and_Health_report.pdf)

The NHS website has an excellent summary of the findings:

[https://www.nhs.uk/news/food-and-diet/the-new-guidelines- on-...](https://www.nhs.uk/news/food-and-diet/the-new-guidelines-on-vitamin-d- what-you-need-to-know/)

</comment><comment><USER5> Dr David Perlmutter is a neurologist who pratices "functional medicine" and makes a lot of articles and videos. The articles on his website often refer to scientific literature and you get get a list of articles by doing a search on his site: [https://www.drperlmutter.com/?s=vitamin+d](https://www.drperlmutter.com/?s=vitamin+d)

You will see a list of articles what Vitamin D does to prevent and heal diseases.

I am taking 15.000 IU/day and have regular blood tests to make sure it stays below 150. I take a lot of other vitamins and minerals and the nodules in my thyroid went from stage 4 to stage 3.

I went to see 2 doctors and they had different opinions: one said it is better to take 30.000 IU every 2 days for better absorption and the other one said it is better to have a stable level. I prefer a stable level.

<comment><USER6> > Dr David Perlmutter

He doesn't seem reliable. [http://nutsci.org/2015/08/03/fact-checking-a- perlmutter-inte...](http://nutsci.org/2015/08/03/fact-checking-a-perlmutter- interview/)

</comment></comment><comment><USER7> Do a search and see a thread from several days ago. Just started, first 6K and later on 10K/day. No conclusions for me yet.

Edit: [https://news.ycombinator.com/item?id=16062697](https://news.ycombinator.com/item?id=16062697)

</comment><comment><USER8> Vitamin D wiki

[https://www.vitamindwiki.com/VitaminDWiki](https://www.vitamindwiki.com/VitaminDWiki)

</comment><comment><USER9> Look up info on cod liver oil. It is high in vitamin D and is how we discovered vitamin D. If you dig a bit, it is known to help a number of conditions, though the Wikipedia article makes it sound like we have hand wavy, vague guesses and nothing definitive. Search for inflammatory conditions, like arthritis, and vitamin D or cod liver oil. Trying to dig up what I saw recently also led me to a piece on PCOS and cod liver. The known benefits of vitamin D seem to not be well advertised or well understood.

The Wikipedia article makes no mention of the fact that people with cystic fibrosis are generally deficient. It is a fat soluble vitamin. People with CF are typically deficient in all fat soluble vitamins and are frequently prescribed supplements for A, D, E and K because of it.

In a nutshell, CF predisposes people to chronic infection and average life expectancy in the US is currently around age 37. Most scholarly articles do not blame the symptoms on things like vitamin deficiency. They basically say having CF causes deficiencies, not that deficiencies cause the symptoms of CF. Which seems pretty dumb to me.

Your bone marrow is an important part of your immune system. Vitamin D plays an important role in bone health. It really shouldn't come as a surprise that a deficiency would undermine your immune system.

It interacts with a bunch of stuff, like calcium and magnesium. Taking a lot of one thing in isolation probably won't have an optimal effect, though there are a lot of confounding factors because if you start eating calcium rich foods because you crave them, you may not realize that you are getting more of other things. Most people don't think of diet that way.

Anyway, if you have reason to think you need this, you should also read up on the other nutrients it is known to significantly interact with. If you need one of them, you like need all of them. If a vitamin D deficiency exists, you likely aren't really on solid ground for things like calcium and magnesium either.

</comment><comment><USER10> A lot of interesting research and opinions has been published during the last ten years. Here are a few links:

"Vitamin D deficiency in undifferentiated connective tissue disease". [https://arthritis- research.biomedcentral.com/articles/10.118...](https://arthritis- research.biomedcentral.com/articles/10.1186/ar2533)

"The Big Vitamin D Mistake". [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5541280/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5541280/)

"For health professionals: Position statement on supplementation, blood levels and sun exposure". [https://www.vitamindcouncil.org/for-health-professionals- pos...](https://www.vitamindcouncil.org/for-health-professionals-position- statement-on-supplementation-blood-levels-and-sun-exposure/)

"The effect of vitamin D supplement on the score and quality of sleep in 20-50 year-old people with sleep disorders compared with control group". [https://www.ncbi.nlm.nih.gov/pubmed/28475473](https://www.ncbi.nlm.nih.gov/pubmed/28475473)

"A Statistical Error in the Estimation of the Recommended Dietary Allowance for Vitamin D". [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210929/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4210929/)

"Dosage of Vitamin D Needed To Achieve 35 to 40 ng/ml (90-100 nmol/L)". [https://www.medscape.com/viewarticle/589256_8](https://www.medscape.com/viewarticle/589256_8) (open in an incognito window to avoid the login screen)

"Vitamin D Status and Acute Respiratory Infection: Cross Sectional Results from the United States National Health and Nutrition Examination Survey, 2001–2006". [http://www.mdpi.com/2072-6643/7/3/1933/htm](http://www.mdpi.com/2072-6643/7/3/1933/htm)

"Additional 4 ng of vitamin D reduced chance of infection by 7 percent". [https://www.vitamindwiki.com/Additional+4+ng+of+vitamin+D+re...](https://www.vitamindwiki.com/Additional+4+ng+of+vitamin+D+reduced+chance+of+infection+by+7+percent+%E2%80%93+June+2011)

"Vitamin D supplementation to patients with frequent respiratory tract infections: a post hoc analysis of a randomized and placebo-controlled trial". [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4553208/](https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4553208/)

I have more links to relevant material, but I have to stop somewhere ;-) As you can see, the body of evidence is growing.

<comment><AUTHOR> Wow that was helpful!

</comment></comment><comment><USER8> Cluster headaches. (CH)

[https://www.clusterheadaches.com.au/forum_posts.php?id=2985](https://www.clusterheadaches.com.au/forum_posts.php?id=2985)

It worked after two or three days.

</comment><comment><USER11> Have you had your levels checked by your doctor? My doctor said I was low for a year, as they came up I started sleeping much better -- can't see many other differences.

</comment><comment><USER12> Two of my family members take Vit D3. It has really helped them a LOT. Going from continuous pain in the legs to negligible pain.</comment>


Does Creativity Decline with Age? - <AUTHOR> http://www.scientificamerican.com/article/does-creativity-decline-with-age/ <comment><USER2> It is an interesting question, but measurement of creativity is a difficult problem in itself.

In proof discovery, as we become more familiar with techniques and tools we begin to rely on them as first principles. As we become older, these approaches become more intuitive through practice. But it is also limiting in a sense that it would be difficult transition to a new field. 20 years of experience in exploration of graphs through probabilistic methods would need to be discarded if an expert was going to begin looking at more algebraic structures. This could be perceived as less creative instead of simply less dynamic.</comment>


Fetchnotes Wants to Get Your "To Do" List Out of Your Head - <AUTHOR> http://allthingsd.com/20111105/fetchnotes-wants-to-get-your-to-do-list-out-of-your-head-to-where-it-will-be-safer/

<comment><AUTHOR> We've opened access to our closed beta for 200 people in the Hacker News community at this link:

www.fetchnotes.com/invite/hackernewsrocks</comment>


Dark market Evolution Marketplace was a scam, $12M USD in Bitcoin stolen - <AUTHOR> https://www.reddit.com/r/SubredditDrama/comments/2zfkee/admins_of_evolution_marketplace_the_current/

<comment><USER2> It really shows the naivety of some of these marketplace users. This is how the "real" drug game works with people getting scammed and robbed everyday. Its not somehow different because its online.</comment>


Facebook tests charging $1 to send messages to strangers - <AUTHOR> http://www.bbc.co.uk/news/technology-20810341

<comment><USER2> I hope that dollar goes 50 cents to Facebook and 50 cents to the poor victim of the message. But what am I saying, this is Mark $uckerberg we're talking about here.

<comment><USER3> TBH that could work. I remember lots of talk in the late 90s about messaging systems where you'd pay the recipient of an email -- there was a not-great- but-good-ideas sf book, I think Earthweb, which had a great moment where a character, feeling crowded by another, bumped up his price for receiving messages from non-contacts to a high amount and realised the other character was in earnest when they paid that amount. Often thought it would make a good startup, though I don't doubt that it's been done and didn't work out for one reason or another.</comment></comment>


Commute Maps Reveal We All Live in Mega-Regions, Not Cities - <AUTHOR> https://www.wired.com/2016/12/mesmerizing-commute-maps-reveal-live-mega-regions-not-cities/ <comment><USER2> 325 million people living in the ruin of the old world and the mega structures of the new one. Mega blocks. Mega highways. Mega Region One.</comment>