Ask HN: Were the latest Wikileaks releases really new information? - <AUTHOR> I keep seeing articles claiming that "cyber security" experts are alarmed (implying surprised) by the latest WL release, and articles claiming that the WL release has re-ignited tensions between Washington and Silicon Valley.
I'm not an expert but as someone with a little bit of familiarity with InfoSec, the latest WL release didn't contain anything surprising. What I've seen so far seems to be information that's already known to be true or widely assumed to be true (e.g., the capability to compromise endpoints). My understanding is efen the source code has been out there for a couple of years.
If I'm right then why would this substantially increase tension between Washington and Silicon Valley beyond a couple news cycles?
It is very possible that my knowledge of InfoSec is so weak that I don't even know when to be surprised... <comment><USER2> Yeah, it seems many security researchers are not impressed:

Bruce Schneier: "One, there is absolutely nothing illegal in the contents of any of this stuff. It's exactly what you'd expect the CIA to be doing in cyberspace."

[https://www.schneier.com/blog/archives/2017/03/more_on_the_c...](https://www.schneier.com/blog/archives/2017/03/more_on_the_cia.html)

Robert Graham: "Most of this dump is child's play, simply malware/trojans cobbled together from bits found on the Internet."

[http://blog.erratasec.com/2017/03/some-comments-on- wikileaks...](http://blog.erratasec.com/2017/03/some-comments-on-wikileaks- ciavault7.html)

the grugq: "The article sadly misses the point about how lame the contents are. The only interesting story is "why" not "what.""

[https://twitter.com/thegrugq/status/839183932763471872](https://twitter.com/thegrugq/status/839183932763471872)

Nicholas Weaver: "The actual data dumped by Wikileaks isn't really that significant, but some entertainment. That it was dumped at all is Yuge."

[https://twitter.com/ncweaver/status/839179237433176065](https://twitter.com/ncweaver/status/839179237433176065)

</comment><comment><USER3> Journalists need to write something and tend to sensationalise everything.

The Intercept has a sane take on the level of "surprisingness": [https://theintercept.com/2017/03/07/cia-has-an-impressive- li...](https://theintercept.com/2017/03/07/cia-has-an-impressive-list-of-ways- to-hack-into-your-smartphone-wikileaks-files-indicate/)

Most of it is just confirmation of things people have known is possible, but never caught in the wild.

</comment><comment><USER4> Here's the latest from Wikileaks

Vault 7: CIA Hacking Tools Revealed

[https://wikileaks.org/ciav7p1/](https://wikileaks.org/ciav7p1/)</comment>


Generalized Information: A Method for Judging Machine Learning Models - <AUTHOR> https://journals.blythinstitute.org/ojs/index.php/cbi/article/view/52 <comment><AUTHOR> Solomonoff induction has long been known to be important in machine learning, where the shorter of two equivalent models is considered "better". This paper extends this in three ways. First, it shows that the data itself can be considered a model of itself. Therefore, an absolute bound for being able to decide that you have a good model from the data is having a model that is smaller than the data. Second, the paper provides a method to quantitatively trade-off accuracy and model size. Third, this methodology is implementation- agnostic, and can in theory be applied to any type of machine learning model.</comment>


Thought Experiment in Mathematics: Gabriel's Horn - <AUTHOR> http://fermatslibrary.com/s/gabriels-horn <comment><USER2> One of the interesting aspects of Gabriel's Horn is that the volume of the rotation of the curve _x → 1 /x_ is finite, even though the area under the curve itself (restricted to single plane) is infinite. Which isn't deeply surprising, but can seem a bit strange if you're new to 3D geometry and calculus.

So being able to quickly navigate relations (and apparent contradictions) like these without pausing to turn the crank and do the integral calculus -- "yeah, the volume is basically the integral of the square of _1 /x_, therefore it's bounded" \-- is really quite crucial.</comment>


Jerusalem doggie DNA database to trace poo offenders - <AUTHOR> http://ca.news.yahoo.com/jerusalem-doggie-dna-database-trace-poo-offenders-130432960.html

<comment><USER2> There was a story about Doggie DNA on This American Life: [http://www.thisamericanlife.org/radio- archives/episode/420/n...](http://www.thisamericanlife.org/radio- archives/episode/420/neighborhood-watch)

</comment><comment><USER3> Sometimes I imagine what the world would look like if the Nazis had won World War II, and this kind of thing is pretty much part of that horrible vision.

<comment><USER4> So where should the line be drawn for using DNA analysis to resolve crimes? I mean how serious should crimes be before DNA analysis should be allowed? Murder, I suppose - but why not dog poo?

Living in the German city with the highest density of dogs in the country, I find dog poo highly annoying. It completely spoils the parks, for example.</comment></comment>


To stop pancreatic cancer from spreading, cut out the chatter - <AUTHOR> https://massivesci.com/articles/cancer-pancreas-trebek-crispr-organ-chip-diseases/ <comment><USER2> The article talks about only 9% surviving five years. That's probably me. I'm three years out from Whipple Surgery (the only fix in 2019) with no sign of a cancer return. My blood is tested and I get CTscans often enough to know everyone by name.

Because I'm a pancreatic unicorn people come to me all the time (a German camgirl recently approached me on her professional account when her uncle was afflicted).

It's very sad and I often feel survivors guilt because I know how it will end for them. Pancreatic cancer is not a pleasant way to die.

Once it has spread it is fatal 100%. When Alex Trebek said Stage 4, I knew his fate was sealed in spite of his positive vibe. Undoubtedly Alex knows too.

How did I get lucky? I got sick one night -- vomit and poop simultaneously. It had nothing to do with my cancer, but it got me the tests that found it. Once it was found I was on the medical conveyor belt.

I am 69 and I've never experience medical care close to this. The care was proactive. They made the referrals and appointments. I just had to show up.

To have my cancer removed (there is no cure) took around a half dozen small procedures where I was put out and an angioplasty where my cardiologist yelled, "Geoffrey, I'm trying to work" when I asked too many questions about the technology. This was followed by a six hour, two surgeon Whipple Surgery (consider small by Whipple standards), a week in the hospital, sixteen staples holding my belly together (and diabetes now that I only half half a pancreas), chemo, radiation and more chemo.

I currently receive NO TREATMENT for cancer. No one you know is luckier. I am playing on house money.

<comment><USER3> thank you for sharing your story it sounds like quite a rough and difficult journey.

<comment><USER2> Worth it. No regrets. I am alive!

</comment></comment></comment><comment><USER4> really surprises me to see how many seemingly effective applications there are with favorable outcomes for ALK7, from just two pages of Google search results.

can basic web scraping produce something meaningful as a recommendation engine for research topics and focus, or are much larger data sets really necessary

</comment><comment><USER5> Is this real or just hype?

We have been so inundated with promises I don't know what to believe.</comment>


2045: The Year Man Becomes Immortal  - <AUTHOR> http://www.time.com/time/health/article/0,8599,2048138-1,00.html2045

<comment><USER2> Dup: <http://news.ycombinator.com/item?id=2201469>

... with much discussion.

<comment><AUTHOR> Oops, sorry about that. Thanks for letting me know.</comment></comment>


Medallion Fund "Stretches Explanation to the Limit," Professor Claims - <AUTHOR> https://www.institutionalinvestor.com/article/b1k2fymby99nj0/Famed-Medallion-Fund-Stretches-Explanation-to-the-Limit-Professor-Claims <comment><USER3> Of the wild theories I've heard to explain the Medallion Fund, my favorite is the "money wormhole." I have no finance qualifications whatsoever -- I just stick my money in index funds -- but I love a good conspiracy theory, so here goes.

The idea is that you have two theoretically unrelated funds that take complementary positions with uneven odds. One sacrifices performance for the other, effectively transmitting money. The trick, of course, is doing this in a manner so that you have plausible deniability with regulators and, more importantly, so that other market players don't grab the money in flight, because that's what an efficient market would do. Still, consistently creating and hiding complementary pairs of good/bad opportunities is a much easier problem than consistently beating the market.

OK, so you've got a way to transmit money from a "sucker fund" to a "winner's fund" with plausible deniability, what do you do with this ability? In reality you'd probably have many "sucker funds," rotate the one that gets sucked, and limit the extent to which it gets sucked, but still, nobody's going to just buy the sucker fund, not for long, anyway. That's how efficient markets work. Well, one major, well-known, boring inefficiency is the principal agent problem: trusting others to invest your money. They can't just steal it, because that's illegal. They can't just invest in a private fund that they just happen to oversee and which just happens to pay them an enormous salary, because that's illegal. However, if they were to invest in one side of a money wormhole, which is designed to look like a perfectly reasonable investment on its own, in exchange for a cut of the proceeds from the other side of the money wormhole, who's to say the theft even happened? Such an accusation would need to untangle the coordination behavior of the money wormhole to make its case, and those inner workings could be made very convoluted indeed, hidden deep inside complementary pseudorandom behavior.

So, in short, the conspiracy theory says it's a heist -- but one that's well hidden under the veil of statistics and plausible deniability. It provides investment managers looking to monetize the confidence placed in them a way to make that happen without getting caught.

It's probably a dumb theory, but it's enough to entertain someone who knows as little about finance as I do :)

<comment><USER4> I'm not sure I understand your suggestion.

So there are (simplistically) two funds, Sucker and Winner. They take complementary positions, and, whichever one wins gets transferred to Winner.

So far so good. Winner is winning every bet, and Sucker is losing every bet.

Now you need people to put money into Sucker, because it has to come out the other end into Winner. How do you convince them to do that? By giving them a cut of Winner's profit?

But that's where things don't add up. How can the Sucker investors and the Winner investors both be making a profit from Winner's funds, if the net money going in and out of the combined funds is equal?

<comment><USER5> People with illegally earned money put their money into a series of short lived sucker funds which exist to funnel money invisibly into the winner fund where it reappears as legitimate investment returns. The dumb version of this with very simple mirror trades which were easy to track has been seen "in the wild" in Russia as carried out by people at DB.

BTW I don't really buy this theory but that's how it would work.

<comment> james-imitative This has been happening in Peru, Uganda, Bolivia and Grenada.

</comment></comment></comment></comment><comment><USER6> > The Medallion fund has been closed to external capital since 1993, and > analysis of the flagship fund's annual returns suggests that significant > distributions are made each year to keep the fund about the same size. For > example, despite the fact that Medallion reported annual net returns above > 29 percent every year between 2010 and 2018, the fund's assets under > management stayed at about $10 billion throughout that period.

This suggests to me that whatever opportunities they're exploiting, they would vanish quickly if they would increase the fund's size (e.g. by compounding returns).

It's worth taking a look at the Numberphile interview with Simons ([https://www.youtube.com/watch?v=QNznD9hMEh0](https://www.youtube.com/watch?v=QNznD9hMEh0)). There he suggests that they're mostly using ML and other relatively simple techniques, with a bit of more advanced math when it comes to predicting how a trade will move the market. This chimes in well with the idea that they have to keep their trades relatively small in order to remain this profitable.

</comment><comment><USER7> As the article hints at, Medallion isn't really an investment fund so much as a pool of capital which is employed in the business of providing tactical liquidity to markets. This is a business where one's competitive advantage rests upon their technological advantage, and Renaissance has been exceptionally adept at building and maintaining that advantage.

The EMH is a theoretical concept that must be tempered to account for the frictions inherent in reality. As my PhD advisor likes to say, "all models are wrong; some are useful." Liquidity is a major friction in real-world markets, and providing liquidity (i.e. a means of moving capital into and out of specific assets) is a service that is compensated accordingly.

In other words, I think Renaissance is simply better than anyone in the world at finding the markets where traders are willing to pay the highest premiums for liquidity, and providing it in a timely and measured dose that ensures they skim the cream off the profit opportunity. There are a limited amount of such opportunities, so they must limit the amount of capital employed to maintain the high rate of return.

<comment><USER8> >Renaissance is simply better than anyone in the world at finding the markets where traders are willing to pay the highest premiums for liquidity, and providing it in a timely and measured dose that ensures they skim the cream off the profit opportunity.

Why are they better?

This still requires the same kind of explanation as 'they are simply better at investing than anyone in the world'

<comment><USER9> >Why are they better?

Possible explanation: there aren't any other firms founded by as great mathematicians as the founders of Renaissance. I saw a quote online attributed to them, something like: "We hire the A-grade mathematicians. Most other firms hire B and C grade mathematicians, and don't even know the A grade exists". This fits my experience, as a D-grade mathematician working in finance. Never heard of significant mathematical prizewinning researchers working at any of the firms I know (lots of maths olympians, but there's a huge difference between winning a maths olympiad and conducting groundbreaking mathematical research).

<comment><USER10> This is only half of it. If you look at all their early employees, Robert Frey/James Patterson have profiles, they were people who knew the maths AND knew how to apply it.

If you look at AHL, they have lots of people who know the maths. They have their own institute at Oxford ffs (the number of people who don't have PHds from Oxbridge there is small), and they have crap performance, year-in, year- out.

It is a typical flaw of human nature to assume that when someone else does something extraordinary that they have some secret knowledge you don't. People on here talking about discrete topology and all kinds of craziness...read the book, one of the big early advantages that RenTech had was they sent some guy down to the Fed to transcribe by hand data that no-one else had. Another example is their stat arb strategy didn't work for years, until they worked out how to get good execution.

I am not saying that it is only this but if you read the book, there are several instances where RenTech try to apply something complex and it doesn't work...and the success they have is after doing something simple, marginally better than anyone else...again, to a certain kind of person, this simplicity is offensive but it is also why most people with technical backgrounds get destroyed by the market.

Also, one thing that is kind of unclear. Simons had almost nothing to do with the perf. By the early 90s, he wasn't personally active anymore (he was spending most of his time on venture capital). The step change was hiring Brown and Mercer (again, two guys who clearly knew the maths but had spent most of their time in industry). Ppl assume that Simons is in the boiler room doing all kinds of crazy technical stuff...he wasn't (most of the time that he ran the fund, it didn't do well at all).

<comment><USER11> "this simplicity is offensive but it is also why most people with technical backgrounds get destroyed by the market"

Figuring out things nobody else can is the hard way to achieve. Figuring out what you _don 't_ know and never betting on it is an alternative way that doesn't require being a genius.

<comment><USER10> Just my 2c, and this is more based on my experience academically, almost no- one figures things out that nobody else can. Innovation is largely a combination of circumstance and timing. Some ideas just have their time. Having unrealistic expectations around this is part of the problem (AHL went in thinking they could just hire a bunch of "boffins"...but they are all average, they just have PHds or whatever).

I don't think it is only knowing what you don't know either. That is a component but people from academia are rarely overconfident, the opposite is usually true (they tend to react badly under pressure). These people have a "circle of competence", they know what they don't know...but they don't really have anything else. All these places are churning massive resources into education, into human capital...is that working? It clearly works in some places and not others. I am just very cautious of the "know what you don't know" argument because it tends towards thinking everyone should be specialists...and I don't think the world works that way.

<comment><USER11> Knowing what you don't know doesn't correspond to a level of confidence or humility, in the way I meant it.

You can have low confidence and discount the value of things you know, while still assuming you know things you don't.

</comment></comment></comment></comment></comment></comment></comment><comment><USER12> I'm no finance expert but doesn't this all seem way too good to be true? At first I was reading this and thinking "how is this not exactly like Bernie Madoff?"

Then they say "The Medallion fund has been closed to external capital since 1993 ... whatever profit they make, they pay out". So clearly it can't be a ponzie scheme?

Still, it seems too good to be true. If financial experts are stumped as well then that also seems like a red flag.

<comment><USER13> I don't think they could keep it up for that long without slipping up at least once. Enron started around the same time and got tripped up in 2001.

If it were a one man investment shop, then maybe it leans more towards fraud. But if they obviously employ dozens of quants, and have obvious hardware outlays, it seems less likely.

It makes a lot of sense to me that they just win a bit more than they lose, play a lot of hands, and treat everything with pretty even money, so they're never overexposed in any one place.

<comment><USER13> I think I have the math right here, but let's say you start with > Capital = $100

1 time a day 5 days a week 50 weeks a year, you're going to take your capital and place that many $1 bets with it.

Let's say you net +1% on your deployed capital ever day.

After 250 rounds, you have $1215 in the bank.

Again, correct the math if I'm wrong. But scale that down to 0.57%, which is the win rate I saw, and they're making ~$600/yr, which is the stated average return.

Seems legit

<comment><USER14> > Let's say you net +1% on your deployed capital ever day.

That's the "..." step in "1 steal underpants, 2 ..., 3 Profit!"

<comment><USER13> Well yeah. But that's what they've figured out.

It's unlikely, but not impossible.

</comment></comment></comment></comment></comment><comment><USER15> Considering the motivations from the various points of view....

As an investor, I'm going to discount anything I can't invest in and just focus on the numbers of the open fund. These closed flagships are a common marketing approach. Like concept cars, they grab attention and help sell the more mundane models that are commercially viable. This article being a good example. Clearly it works in this role and I can understand why they would do this.

As a PM, I'd always prefer to have my own capital in a closed flagship where I can cherry pick the best strategies, charge myself less fees and make use of the latest tech and R&D subsidised by the open funds. My reporting requirements are much less onerous so there is plenty of scope to massage the numbers too.

As a regulator, I'm underfunded and politically motivated to focus on high profile cases where large numbers of everyday public (voters) have been ripped off. I have little time for institutional investors who should know better and so closed funds trading internal money are way down my list.

As a regTech founder, I've seen plenty of examples of creative methods to get around regulations. Or they just being blatantly ignored. Compliance are usually colluding and helping coverup. Insider trading is one of the higher risk methods, to get these sort of returns it would be easier to simply get access to the "dumb" order flow from the open funds. Using lots of fancy looking algos etc. to hide simple front running.

Given they are closed and trading their own money we are only going to know what they choose to disclose. Personally, I'd bet on this being legit, given the figures while great, aren't impossible given the privileged position they have in terms of access, information, tech etc. And the existing distribution of performance for low capacity HFT strategies. People abusing the system would be making even more and keeping quiet about it.

</comment><comment><USER16> My long standing hypothesis on Medallion is that they figured out how to apply gauge theoretic techniques to financial markets. This fits with Simons work that he did before he founded the fund. The fact that gauge theory is applicable to for example currency trading is folk knowledge in the Havard, Princeton, IAS circles (here is for example the lecture notes of a popular lecture by Maldacena that uses currency trading as an example of a gauge theory [https://arxiv.org/pdf/1410.6753.pdf](https://arxiv.org/pdf/1410.6753.pdf)).

In currency trading example the curvature $F$ of the gauge connection $A$ vanishes precisely, when there is no arbitrage opportunity. Now what Chern and Simons discovered is a differential form K ([https://en.wikipedia.org/wiki/Chern–Simons_form](https://en.wikipedia.org/wiki/Chern–Simons_form)), which when taken as the action S(A) of a gauge field $A$, has a corresponding field equation $F = 0$ (i.e. in the finance case = no arbitrage). In these equations time does not enter, however there are several ways in which on can incorporate time into the picture (for a naive example see also the lecture notes). Assets in this framework live in associated vector bundles of the principal bundle defining exchange rates.

My speculative assertion is that it is possible to identify "topological invariants" which can be computed by sequences of trades, i.e. parallel transport along the gauge connection and that those can have provably positive expected return. The fact that the fund is limited to a small amount of invested capital might be related to the fact that the strategies require measurements, that would have self-interactions if they were too large.

<comment><USER17> > My long standing hypothesis on Medallion is that they figured out how to > apply gauge theoretic techniques to financial markets.

No. Listen to the Talking Machines podcast with Nick Patterson (who was a senior VP in research at RennTech for a long time). To paraphrase he says that the vast majority of their strategies are no more than simple linear regression. The challenge is that even though regression is conceptually simple it still takes smart people to answer questions like "what should you be regressing", or "should you apply any transform" or "how should you clean your data" or "do you understand the process well enough to realize when results are obviously unrealistic".

The thing that makes a firm like Renaissance a league above a firm like Two Sigma is the same thing that makes Two Sigma a league above a firm like Winton. It's not mathematical gnosticism, it's plain old operational excellence. It's things like expansive reliable curated datasets, deep expertise on market structure, good execution systems, powerful research and backtesting software, good access to markets, economies of scale, talented practitioners, and excellent organizational management.

<comment><USER16> The assertions you make are not necessarily in contradiction to what I am saying. In discretised form most of the formulas I'm talking about boil down to simple linear algebra with unknown parameters. You can then use essentially linear regression to find those parameters, based on observed market data and trades you are making.

So I was talking about the "what should I be regressing" and "transformation" (you can use gauge theory to adjust for inflation and changes in exchange rate in non-obvious ways) part. There is no question that having access to enough data and operational excellent are a complementary component.

</comment></comment></comment><comment><USER18> Seems to me there's an obvious flaw in the efficient market hypothesis. It states that, essentially, you can find no sustainable edge because the market rapidly reacts to information. Meaning, if there is some information relevant to expected investment performance, investors will immediately act on it, extinguishing the information advantage.

But what if the opportunity is some kind of abstract pattern that doesn't make any sense to humans? You can hardly say that people would efficiently react to patterns that are entirely uninterpretable to them. So it leaves the door open for these sort of statistical artifacts - abstract side-effects or unintuitive properties of the overall system - to provide a sustainable source of alpha.

<comment><USER17> The efficient market hypothesis is a lot like Newtonian gravity. It's not a perfect theory, but it's a pretty close approximation that pretty much covers most any domain outside very exotic conditions.

If somebody comes up to you, a random Joe Schmoe, and tells you they have an investment that consistently beats the market on a risk-adjusted basis. Well... You can pretty much guarantee that they're full of shit.

Even just analyzing papers that passed the rigor of academic peer review, the sizable majority of market anomalies fail to replicate on an out-of-sample basis[1].

Like Newtonian gravity, to the extent that deviations exist it's either of very small magnitude or in very exotic conditions. The market anomalies that do exist, like the HML value effect or momentum effect, don't substantially improve the optimal portfolio and go through very long periods of deep drawdowns.

Or in the realm of the very exotic, there are small teams, in aggregate making up much less than 0.1% of the market, that do consistently out-perform. There common characteristics of these strategies are that they have tightly limited capacity, involve a huge up-front investment in technology and expertise, almost never offer to manage outsider money, and are extremely hard to replicate even by other experts.

[1] [https://www.nber.org/papers/w23394](https://www.nber.org/papers/w23394)

<comment><USER18> If the value and momentum and size factors are legitimate, then it stands to reason that there could be other more obscure, less intuitive and more profitable anomalies. Are those limited to very small magnitudes? Maybe, but I don't see why that should necessarily be so. It seems like EMH is predicated on the idea that the only investment opportunities are ones that map to human reasoning.

</comment></comment></comment><comment><USER19> There are two simple explanations for this:

\- somebody lies.

\- one out of hundres of thousands may seem (and be) incredibly lucky for quite long. Until it isn't.

With the amount of transparency involved in this case for all I know they could be just laundering money for mafia by doing too many transactions that anyone could ever audit and claiming profit on them.

It's way easier explanation than "random walk is not random" and even easier than "they got 30 years of luck on random walk where second best got 5 or sth." which still is not so implausible as the first one.

</comment><comment><USER20> The explanations given in the article are good ones and are said before. Even in HN [https://news.ycombinator.com/item?id=13033733](https://news.ycombinator.com/item?id=13033733)

They have a strategy that does not scale beyond certain limit and they keep the fund tightly within those limits.

</comment><comment><USER10> Asking a finance professor about a hedge fund is like asking someone from the 1850s how a car works. Wrong tools. Wrong knowledge. Wrong way of thinking. Like he is talking about EMH...lul, it is just fiction.

</comment><comment><USER21> Madoff has unexplained gains year over year as well based on new methods.

The explanation as offered triggers me but I haven't looked into it in detail.

For Madoff the gains were truly magical as there was little evidence he actually traded.

Interestingly, Epstein supposedly has magical gains as well with little evidence he traded.

</comment><comment><USER22> How big are the external funds? If they are large enough, it wouldn't be hard (quantitatively) to shave a percent or two off of their returns in bad medallion years to keep the winning streak alive.

The inflows inspired by the mystique shrouding medallion alone could make this easy

<comment><USER23> There are strict regulatory obligations to the outside investors that would make that transfer illegal.

<comment><USER12> illegal, sure. But let's say there are some very bad actors involved... would it be possible? People have been known to do illegal things in finance and have tried to cover their tracks in the past.

<comment><USER10> Nope. This used to be quite common in the 80s (you would run a public fund, and allocate losing positions to pension funds) but doesn't happen today.

</comment></comment></comment></comment><comment><USER24> These comments are almost all a dumpster fire of clueless speculation. I normally suggest people read Hacker News comments, but this is embarrassing. The uninformed ramblings about winner v. sucker funds, or assertions of insider trading are sad.

The only factual comment here was by MR4D about how IRA accounts can impact wash sale rules, as would all accounts you own in prime brokerage.

Hint: maybe there is a high probability the author of this paper doesn't know what he is doing. It wouldn't be the first time: [https://www.latimes.com/business/hiltzik/la-fi-hiltzik- corne...](https://www.latimes.com/business/hiltzik/la-fi-hiltzik-cornell- verizon-20190219-story.html)

Or maybe he's selling a competing product: [https://cornell- capital.com/about-2](https://cornell-capital.com/about-2) but can't compete.

</comment><comment><USER25> the magic $10 billion number capping the size of the fund also seems curious because why would it be impervious to change over a period of 25 yrs in which the scale of trading volumes, market capitalization, etc. have all grown?

also, groups of people don't perfectly share ideas, outlooks, etc. especially over time. Humans tend to argue, debate, tug of war. If this task (the puzzle they are solving) requires a team of people, the 1st tiers and 2nd tiers, again, how does that translate to a steady $10 billion "working capital" portfolio and steady returns over 25 yrs, this money sucking tick parasitically attaching itself to a stochastic market that has undergone vast changes, but itself staying so consistent?

<comment><USER26> I don't believe the size has in fact been constant. I remember reading years ago that Medallion was capped at around $5 billion.

</comment></comment><comment><USER27> >Robert Mercer, the former co–chief executive of Renaissance Technologies, allegedly told a friend that Medallion was right 50.75 percent of the time when it came to its millions of trades — adding that "you can make billions that way."

Totally not a finance guy, but how can this be true with drawdowns? The losses are worse than the gains- if you had $100 and take a 33% loss, it would take a bit more than a 50% gain to get back to where you were. 50% loss and you'd need to double your money to get back, etc.

So.... with drawdowns, don't you need to be right more than 50.75% of the time?

<comment><USER28> No, you bet 100 x $1, win 50.75% of the time, so you make $1.5 in profit. Now repeat 1bn times.

<comment><USER27> But if you're losing (undefined amount of money) on 49.25% of the trades....

</comment></comment></comment><comment><USER29> If it looks too good to be true, it usually is.

</comment><comment><USER30> I have a much more simpler (not based on any evidence) theory which kind of explains the founders' alignment with Russian interests.

Most of medaillon's returns are made with insider trading from hacking. The data science part is just come up with explanations that are complex enough and not easily disproven (e.g. find a correlating variable and explain that you had a magic algo that derived prediction from that correlating variable after the fact).

It's a perfect crime, but you would need to setup crazy incentive structures for all the team members and their immediate family to keep it secret... and that's what they are doing.

I would guess they used Russian hackers for the job, given that the fund is heavily funding Russian political interests in western countries.

I would recommend the SEC to dive really deep with appropriate expertise into this magic fund and double-check this simple theory of mine. Just for national security reasons alone.

(note again: this is a conspiracy theory and not evidence or fact-based)

<comment><USER31> Keeping something like this secret is... Hard.

<comment><AUTHOR> NSA will know about this in a heartbeat

</comment></comment></comment><comment><USER32> The hedge fund business is and has always been based on insider trading. Ockham's razor looms large over this conversation.

<comment><USER33> If that were true, you'd expect that as a group they would have better risk adjusted returns. I personally have not witnessed any insider trading in the industry and suspect it quite rare nowadays. It was more common in the past, though.

<comment><USER14> Some hedge funds are good at it, and some are bad it. The existence of a winnable game doesn't mean that everyone who attempts it will succeed. Especially since the game is competitive, so if I have better insider information than you, I can take from you you all the money you take from the non-insider traders.

</comment></comment></comment><comment><USER34> What are they talking about? Rentech went into the dumpster.

[https://seekingalpha.com/news/3300183-rentech-to-delist- from...](https://seekingalpha.com/news/3300183-rentech-to-delist-from-nasdaq- sharesminus-26)

<comment><USER35> Doesn't look like that's the same Rentech: [https://seekingalpha.com/symbol/RTK](https://seekingalpha.com/symbol/RTK)

"Owns and operates wood fibre and nitrogen fertilizer businesses"

</comment></comment><comment><USER36> Investing is like tech, its winner take all. This is like being surprised that Google dominates search

<comment><USER37> No, it's the exact opposite. The limit of how much money you can put through a given strategy mean there is an antieconomy of scale that encourages lots of small firms.

<comment><USER36> this is the complete opposite of true. I work in the industry, too busy to write up a response. But basically, the top five hedge funds are making most of the returns and are attracting most of the capital. the industry is consolidating

<comment><USER37> It's certainly plausible that the few best funds are outliers in terms of return. And I'm sure they _attract_ a lot of capital in the sense that a lot of people want to invest in them, but that doesn't mean they allow it. As far as I can tell, the top 5 hedge funds by AUM have around 10% of the whole sector's capital. In comparison, the top 5 tech companies in the S&P500 have market caps summing to around 10% of the whole index (not just the tech sector).

</comment></comment></comment></comment><comment><USER38> The nasdaq 100 leveraged x3 on a daily basis returned like 50%+ YoY during the last decade. And that's an index. Buffet himself said he could return 50% YoY consistently with a small(ish) amount of money. (He manages like half a trillion)

there's no reason why with $10B trading all asset classes one can't return 70% YoY. You must note that the fund is capped, the execution costs are incredibly low and that over the last decade it returned less than 70% YoY.

<comment><USER39> 3x levered nasdaq 100 would give you an annual volatility of at least 30-50%, occasionally much much higher. The crazy thing is that medallion presumably achieved this with a very small volatility and no significant drawdowns. Nasdaq 100 was down -42% in 2008, so levered 3x you would be out of business.

<comment><USER33> 3x leverage ETFs rebalance daily, so you would still have some money since the biggest single day loss of Nasdaq is ~10%.

<comment><USER40> Incidentally this also means that if you buy the etf on day 1 and the underlying is at 100, on day 2 the underlying falls to 99 then on day 3 goes back to 100 then you would be left with less than 100% of what you started with (assuming perfect tracking and no fees).

<comment><USER41> Yes, this is called "decay", and it's why you don't want to hold a leveraged fund over any long period.

<comment><USER33> According to my models there are very few market environments in which you would make less money with a leverage ratio of 2.5x. In order maximize return your leverage should be:

Expected Return/Expected Variance

For example even if the expected return is 1% and the vol 5%, the ideal leverage ratio for maximizing return is 4x!

In short, a 2-3x leveraged ETF is an excellent investment and should outperform the index in almost all market conditions. It's when your leverage ratio goes over 5x that you start to have major problems a lot of the time.

<comment><USER41> I can appreciate this, but the fund decay actually has nothing to do with leverage.

All "leveraged" ETFs (to the best of my knowledge) are synthetic - <USER2> achieve their "leverage" using derivatives, not by borrowing. These derivatives are not free, and like an option, can expire worthless. That's how the value in these ETFs evaporates over time, regardless of how the market performs.

<comment><USER33> Of course not "regardless" of how the market performs, take a look at UPRO over the last 2/3 yrs. But those are in theory reasonable concerns, however empirically most leveraged funds have performed as promised relative to their benchmarks (with a couple notable exceptions I admit). The entire point of derivatives (as suggested in the name) is that they inherantly bear an underlying relationship to their underlying security.

If you look at UPRO, its daily returns almost exactly track 3x of SPY. There's no long-term "decay", unless you are referring to volatility drag. VIX etfs are the notable exception, in that they do suffer from persistant negative carry.</comment></comment></comment></comment></comment></comment></comment></comment>


Is Agile for everyone? - <AUTHOR> http://www.andybrandt.net/653/is-agile-for-everyone <comment><USER3> I do not necessarily disagree completely, but I am a bit more optimistic about this. It is definitely possible (I've done it) to do agile projects even when all the team members are not self-motivated, provided there are at least one or two alpha-geeks that are. So I don't think it is the case that you need to stack the whole IT industry with hackers in order to have agile work.

But particularly as to this:

 _Good people tend to gravitate towards interesting projects, new languages, challenging problems – stuff they can brag about to their peers over beer. Many software projects don't meet this description, but still have to be done._

I think this is part of the problem. Structuring a project so that it _does_ have some interest to a good software engineer is one of the challenges of managing an agile project. You would be amazed what people find interesting - I know tons of great developers that are not new-technology- or brag-driven (though none of them would agree to work on an "evil" project I don't think). The trick is to get the people first, and structure the project later - <USER2> is why this fails a lot in the real world, when personnel ramping-up happens after the project has been defined.</comment>


Handling blocking IO in Go - <AUTHOR> http://www.kanwisher.com/2012/07/19/handling-blocking-io-in-go/

<comment><USER2> I don't know why author thinks that blocking IO is a "One of the most frustrating things when you start with go" because from the very beginning of leaning Go most articles, tutorials and books tells that golang uses goroutines to achieve concurrency. I've started learning golang a few weeks ago and the fact that all IO is blocking was the most nicest part of the language. I mean you dont need to wrap your code in deeply nested callbacks as you do in node.js, and your code is always look straightforward. In fact IO is only _looks_ blocking, AFAIK, go uses epoll/kqueue under the hood for IO operations.

</comment><comment><USER3> Good start, but needs more explanations I think. If I was totally new I wouldn't understand what this stuff is.

Actually, I _am_ totally new, but I've seen talks and read various tutorials, so I can understand the concepts presented here.

I think this stuff is really important and people really should become aware of how do handle blocking IO in Go. Maybe then people will finally realize that things like Node (and Twisted, and Tornado, etc) are not really a good way to handle blocking IO after all.

I'm very excited about Go, and I don't understand why it's not catching up yet with the startup crowd.

<comment><USER4> > If I was totally new I wouldn't understand what this stuff is.

If you were "totally new" and didn't even know about go and goroutines, why would you even click on that link _and expect to understand what it's talking about_?

> Maybe then people will finally realize that things like Node (and Twisted, > and Tornado, etc) are not really a good way to handle blocking IO after all.

Wow, that's one big strawman to set on fire.

Of course evented systems "are not really a good way to handle blocking IO", nobody's ever claimed that, quite the opposite. There's a reason why the IO layers have to be provided by the evented runtime or patched (and replaced by non-blocking versions) on the original one. Here's how you handle blocking IO in an evented system: you bloody fucking don't.

For the outlined case, a node user would use child_process[0] which is non- blocking and provides all the tools you should need (I'm sure there are libraries out there providing an even nicer interface to sub-processes).

[0] <http://nodejs.org/api/child_process.htm>

<comment><USER3> > If you were "totally new" and didn't even know about go and goroutines, why > would you even click on that link and expect to understand what it's talking > about?

That doesn't even make any sense.

You're objecting about why a person would read an introduction to a language/platform if they're not familiar with it.

> Of course evented systems "are not really a good way to handle blocking IO", > nobody's ever claimed that, quite the opposite.

Oh really? That's _all_ Node.js claims to do.

What's Node's tagline? "Evented I/O".

Node's only purpose is to "fix" blocking I/O.

Go on, read what the homepage says:

<http://nodejs.org/>

 _Node.js is a platform built on Chrome's JavaScript runtime for easily building fast, scalable network applications. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient, perfect for data-intensive real-time applications that run across distributed devices._

Notice how it doesn't sell itself as "javascript on the server side" or anything like that.

It sells itself on two points:

\- A platform that runs on top of Google's V8 (fast)

\- It fixes I/O blockage by using events and callbacks (aka evented I/O)

In other words, it's all about scalable backend services.

If you think Node's approach is a good way to handle blocking IO, good for you.

I happen to think it's a terrible approach. And I think there is a much more viable alternative out there for backend developers who have performance concerns: whether it's related to IO blockage, or the general slowness of interpreted languages.

<comment><USER4> > You're objecting about why a person would read an introduction to a > language/platform if they're not familiar with it.

No, this is _not_ an introduction to a language/platform, this is an introduction to a very specific problem/detail of a language/platform.

> Node's only purpose is to "fix" blocking I/O.

By _getting rid of it_.

> If you think Node's approach is a good way to handle blocking IO, good for > you.

Are you trying for a daftness award?

<comment><USER3> > Are you trying for a daftness award?

No, you are nitpicking on the word "handle". Alright, let's change that to "deal with the problem of IO blockage". Happy now?

You know full well what I mean, and I can't interpret your attitude as anything other than trolling.

</comment></comment></comment></comment></comment><comment><USER5> Source: [https://groups.google.com/forum/?fromgroups#!topic/golang- nu...](https://groups.google.com/forum/?fromgroups#!topic/golang- nuts/MN_W1_oAFrs)

Not sure what the original expectation was. How else would an operation, that by definition continuously writes to stdout until interrupted be anything _but_ blocking... until the process was killed?</comment>


Security Issue, February 2019 - <AUTHOR> https://support.500px.com/hc/en-us/articles/360017752493-Security-Issue-February-2019-FAQ <comment><USER2> > A system-wide password reset is currently underway for all users, > prioritized in order of potential risk, and we have already forced a reset > of all MD5-encrypted passwords.

ERRRR big red flag here.

</comment><comment><AUTHOR> They're sending out emails (which is what prompted me to submit this), including:

> We recognize that you have chosen to deactivate your account, however, some > of the information that you provided us with when filling out your user > profile may have been compromised (listed below).</comment>


Show HN: IPFessay – Publish uncensorable essays to IPFS - <AUTHOR> https://gitlab.com/stavros/IPFessay <comment><USER2> For context: IPFS is a distributed content-addressable storage system that is web compatible. If you want to learn how to use IPFS directly (i.e. without using and trusting a public gateway), start with the Getting Started guide: [https://ipfs.io/docs/getting-started/](https://ipfs.io/docs/getting-started/)

(I'm not the OP, but) If you want to use IPFessay without installing IPFS, you can use it on my hardbin.com public writable gateway (most gateways are not writable) at:

[https://hardbin.com/ipfs/QmQUrYJ6sCSDV6duC4Db7U8NXZG3DpNrVF7...](https://hardbin.com/ipfs/QmQUrYJ6sCSDV6duC4Db7U8NXZG3DpNrVF77XtcrbWYiJY)

and the sample essay:

[https://hardbin.com/ipfs/QmWFSuFDaApxK6DaBzEXdSXhb325fQY1hHn...](https://hardbin.com/ipfs/QmWFSuFDaApxK6DaBzEXdSXhb325fQY1hHn7Jc32DS77UK)

And if you want your resulting essay to stick around, you can pay a service like ipfsstore.it to pin it for you: [http://ipfsstore.it/](http://ipfsstore.it/)

EDIT: It doesn't actually work on my public gateway because it uses the IPFS node API instead of the standard writable gateway API. I'll submit a pull request to make it use the normal writable gateway API tomorrow.

EDIT2: StavrosK has updated it to use the gateway API! I updated the hash link so it does now work :)

<comment><AUTHOR> Hey, thanks! Can we make it use your API by default, or perhaps as a fallback?

I'm not sure what the difference between the Node API and the Gateway API is, I had to hack around this by reading the js-ipfs code. If there's a better way to interface with the gateway without including the entire js-ipfs 2.5 MB file, please let me know.

<comment><USER2> It's not my API, it's the standard IPFS writable gateway API, documented here: [https://discuss.ipfs.io/t/writeable-http- gateways/210](https://discuss.ipfs.io/t/writeable-http-gateways/210)

And there's an example usage here: [https://github.com/jes/hardbin/blob/master/js/hardbin.js#L25](https://github.com/jes/hardbin/blob/master/js/hardbin.js#L25)

There is currently a bug which means you first need to DELETE the filename (in my case "content") that you are trying to replace if it already exists, but apart from that it is straightforward.

I started fixing the bug here [https://github.com/ipfs/go- ipfs/pull/3931](https://github.com/ipfs/go-ipfs/pull/3931) but ran out of steam - hopefully I'll continue soon.

<comment><AUTHOR> Sorry, I meant your gateway. Thanks for the documentation, I had no idea this existed. I'll have a look, but if you already know how to use this API and would like to issue a MR, I'd be grateful.

EDIT: I have updated it to use the Gateway API, thanks for letting me know.

<comment><USER2> Excellent! I updated my comment.

> Here's the permanent, immutable IPFS link to your essay: > > /ipfsQmZo1JgakWB2ZYrA75P3b7eZxvwP1vKpDw2y7WJJhj8j7p

There's a "/" missing in here :)

<comment><AUTHOR> Fixed, thanks! I have taken the liberty of linking things to hardbin.com, including the IPNS link:

[https://hardbin.com/ipns/QmSWnBwMKZ28tcgMFdihD8XS7p6QzdRSGf7...](https://hardbin.com/ipns/QmSWnBwMKZ28tcgMFdihD8XS7p6QzdRSGf71cCybaETSsU/)

Thanks for your help!

</comment></comment></comment></comment></comment></comment><comment><USER3> IPFS isn't "uncensorable" and shouldn't be promoted as such. Nodes must openly host your content for it to be available in the network.

<comment><USER2> Everybody on the network is a node. As long as anybody who is hosting the content (even implicitly, in their cache) is online, it remains reachable.

<comment><USER4> Doesn't that create some legal issues?

By viewing some of these IPFS links higher up in the thread am I "distributing" their contents in the eyes of the law?

<comment><USER2> Unlikely. Even if you saw something "bad", you have a perfectly reasonable explanation. I don't think it'll be a problem.

<comment><USER5> I mean, people have had doors kicked down for operating Tor exits.

[http://www.npr.org/sections/alltechconsidered/2016/04/04/472...](http://www.npr.org/sections/alltechconsidered/2016/04/04/472992023/when- a-dark-web-volunteer-gets-raided-by-the-police)

> Seattle police spokesman Sean Whitcomb says the department understands how > Tor relays work, and they knew Robinson was a Tor host.

> "Knowing that, moving in, it doesn't automatically preclude the idea that > the people running Tor are not also involved in child porn," Whitcomb says. > "It does offer a plausible alibi, but it's still something that we need to > check out."

</comment></comment></comment></comment></comment><comment><USER6> Starting from May 18, 2018, the EU will enforce "the right to be forgotten".

Will it be illegal to run IPFS servers within the EU?

<comment><USER7> Running IPFS by itself does not mean you're hosting all content in the IPFS network. Rather, you need to explicitly agree to share content for it to be served.

If you are hosting certain content, I guess someone can approach the node owner and request you to take the content offline. But nodes outside EU could still host the content.

<comment><AUTHOR> > Rather, you need to explicitly agree to share content for it to be served.

Hmm, how do you do that? My node is sharing all content in its cache by default, AFAICT.

<comment><USER7> Accessing content puts it in the cache, which would be removed when `ipfs repo gc` is run (or automatically at certain points). However, content that is pinned will survive the GC, which is the explicit approval of content that should be seeded for as long as the node exists (or until you run `ipfs pin rm $HASH && ipfs repo gc`)

<comment><AUTHOR> Ah, by "all content" you meant " _all_ content", I see. Yes, but "explicit agreement" is just "loading the content", so it's not _very_ explicit.

</comment></comment></comment></comment></comment><comment><USER8> Does it enforce Cross-Origin Policies?

Since the addresses are just path components and not subdomains this may be an issue.

It's just a feeling this might get relevant, especially with PUT/POST write API and so on. Even just running on localhost might pose issues.

<comment><AUTHOR> Does what enforce it? The browser does, that's why you need to set CORS on your local node for it to work. What's the scenario you're thinking of?

<comment><USER8> Different authors and documents are considered part of the same domain since the tokens are just path elements. The whole IPFS gateway might be considered one big domain.

So javascript in an essay can access/put the API.

I didn't really look into it too much, just wanted to give a pointer that this might need to be considered. In the sense that it might have security implications.

<comment><AUTHOR> No, you're right in this, if you have it enabled, any app can put stuff to your endpoint. The people of IPFS are working on fixing this problem, but I personally think that the risk is low enough. Maybe I'm just not creative enough in thinking about attacks, though.

</comment></comment></comment></comment><comment><USER9> These are only as "uncensorable" as any other IPFS object, right?

Which is to say, not very. IPFS isn't even encrypted right now.

<comment><AUTHOR> Yes, they are.

Although, how is IPFS censorable? As long as you can connect to a node, you can retrieve its content. You'd have to prevent connection to the entire network to censor something (minus the encryption that you correctly stated).

<comment><USER9> > You'd have to prevent connection to the entire network to censor something

That's not necessary if you can perform deep packet inspection or a MITM attack on IPFS traffic, which there's nothing to stop you doing right now.

<comment><AUTHOR> There's nothing to stop you _now_ , but that's just because there's no encryption. Once encryption is implemented, this will be harder/impossible to do (especially since it's trivial for a client to verify that the content it got for some hash is correct).

<comment><USER9> Perhaps it'll be a great system if or when that happens. But it is a stretch to call it "uncensorable" when it isn't, and clearly hasn't been designed for it.

<comment><AUTHOR> I don't know, when censorship entails deep packet inspection between _all pairs of nodes on the internet_ , I'm good with "uncensorable" as a shorthand for "practically impossible to censor unless you control the entire internet".

<comment><USER10> You don't need to control the entire Internet.

See: Firewall of China.

You only need to control the big fiber paths of which there aren't many. Of just in general the exit and entry points between countries.

The Internet is not an open mesh.

<comment><AUTHOR> Even with the firewall, you can't control what's going on inside of China, and you only need one node inside to fetch something from the outside, and that's it.

<comment><USER10> You can certainly control incoming traffic to china.

And once inside, you are subject to extreme government spying which can easily mean that everyone who gets a hold of unallowed content "vanishes".

Furthermore, this is an issue not even encryption can fully solve, you still need to do a key exchange which means you need to either find a source of trust in the other side via sidechannels (aka PKI, TOFU or similar) or you allow everyone to easily intercept the data without much effort (aka skip- verify on TLS)

The internet in china is not lacking in surveillance and censorship.

This is an issue that even Tor is still chewing on a lot, afaik plain Tor connections in China are not very welcome and even blocked.

So unless you claim that your crypt and security is better and more battletested than Tor, I'd say IPFS is anything but censorship resistant.

<comment><USER7> You can always have a USB-stick with the IPFS repository on it, and travel to china, setup your node within and start sharing it. People can trust the data is correct as they request the data based on the hash.

Another way to distribute once the data is within China is to do it via private networks built with cjdns or similar

<comment><USER10> This assumes that your USB stick doesn't get checked on the airport.

Secondly, people can exactly not trust the data because there is no way to trustlessly get those hashes to begin with from within IPFS and you get back to normal HTTP which... is censorable.

Lastly, this also assumes that the one running the node is not arrested either.

>Another way to distribute once the data is within China is to do it via private networks built with cjdns or similar

Which again, assumes cjdns and other methods do not get people into possibly lethal trouble or aren't censored/outright blocked to begin with.

edit: You also forgot about the government blocking IPFS within their own country.

</comment></comment></comment></comment></comment></comment></comment></comment></comment></comment></comment><comment><USER11> So, is IPFS more than Freenet + a fresh coat of 2017 paint?

<comment><USER7> Yeah, I would say the biggest difference is in how content gets reshared. With Freenet, content gets reshared by nodes as soon as it enters the network, and users are unaware of what content exactly they are sharing.

While IPFS is not implicitly pushing content but nodes needs to request content and then reshare it.

<comment><USER12> This is very important, because it removes some of the most prevalent criticisms about Freenet. If you don't agree with some content (re: legality or any reason you wish), don't access it/share it. Freenet doesn't give you that choice, but IPFS does.

</comment></comment></comment><comment><USER13> I don't think "essays" are in great danger of being censored on the internet.

<comment><USER14> Germany has been raiding homes over Facebook posts. Right now they're considering a law to fine social media companies who don't remove certain posts within 24 hours. [https://www.nytimes.com/2017/06/20/world/europe/germany-36-a...](https://www.nytimes.com/2017/06/20/world/europe/germany-36-accused- of-hateful-postings-over-social-media.html)

Twitter already blocks known Nazi accounts in certain countries. [https://twitter.com/map/status/869594711991963653](https://twitter.com/map/status/869594711991963653)

<comment><AUTHOR> And you don't have true freedom of speech unless you allow speech you disagree with... I wonder how we got to the current "speech is free unless you're someone we don't like" society, and I say this as someone who dislikes Nazis.

I want to write a post on tolerance one of these days, and how tolerating things you like isn't really tolerance.

<comment><USER15> > how tolerating things you like isn't really tolerance

This idea was presented in section 1 of

[https://slatestarcodex.com/2014/09/30/i-can-tolerate- anythin...](https://slatestarcodex.com/2014/09/30/i-can-tolerate-anything- except-the-outgroup/)

<comment><AUTHOR> Yes, I had read that, it's a very interesting piece. I had some more thoughts on the matter, but that article is definitely worth a read.</comment></comment></comment></comment></comment>


Polyvore in-depth (fashion vertical: $40CPM) - <AUTHOR> http://www.newyorker.com/reporting/2010/03/29/100329fa_fact_jacobs?printable=true

<comment><AUTHOR> Among many other things, I thought it was interesting to 'anchor' the fashion vertical around $40CPM as a comparison to LinkedIn (business/recruiting vertical), reportedly around $80CPM.

</comment><comment><USER2> This is really a remarkable site. How is it that I hadn't heard of it before now? None of the women I Follow on Twitter have ever mentioned it.</comment>


Low-Latency Video Processing Using Thousands of Tiny Threads [pdf] - <AUTHOR> https://www.usenix.org/system/files/conference/nsdi17/nsdi17-fouladi.pdf <comment><USER2> Co-author here -- thanks for linking to our paper. The basic theme is that (a) purely functional implementations of things like video codecs can allow finer- granularity parallelism than previously realized [e.g. smaller than the interval between key frames], and (b) lambda can be used to invoke thousands of threads very quickly, running an arbitrary statically linked Linux binary, to do big jobs interactively. (Subsequent work like PyWren, by my old MIT roommate, have now done (b) more generally than us.)

My student's talk, which is worth watching and has a cool demo of using the same system for bulk facial recognition, is here: [https://www.usenix.org/conference/nsdi17/technical- sessions/...](https://www.usenix.org/conference/nsdi17/technical- sessions/presentation/fouladi)

Daniel Horn, Ken Elkabany, Chris Lesniewski, and I used the same idea of finer-granularity parallelism at inconvenient boundaries, and initially some of the same code (a purely functional implementation of the VP8 entropy codec, and a purely functional implementation of the JPEG DC-predicted Huffman coder that can resume encoding from midstream) in our Lepton project at Dropbox to compress 200+ PB of their JPEG files, splitting each transcoded JPEG at an arbitrary byte boundary to match the filesystem blocks: [https://www.usenix.org/conference/nsdi17/technical- sessions/...](https://www.usenix.org/conference/nsdi17/technical- sessions/presentation/horn)

My students and I also have an upcoming paper at NSDI 2018 that uses the same purely functional codec to do better videoconferencing (compared with Facetime, Hangouts, Skype, and WebRTC with and without VP9-SVC). The idea is that having a purely functional codec lets you explore execution paths without committing to them, so the videoconferencing system can try different quantizers for each frame and look at the resulting compressed sizes (or no quantizer, just skipping the frame even after encoding) until it finds one that matches its estimate of the network's capacity at that moment.

One conclusion is that video codecs (and JPEG codecs...) should support a save/restore interface to give a succinct summary of their internal state that can be thawed out elsewhere to let you resume from that point. We've shown now that if you have that (at least for VP8, but probably also for VP9/H.264/265 as these are very similar) you can do a lot of cool tricks, even trouncing H.264/265-based videoconferencing apps on quality and delay. (Code here: [https://github.com/excamera/alfalfa](https://github.com/excamera/alfalfa)). Mosh (mobile shell) is really about the same thing and is sort of videoconferencing for terminals -- if you have a purely functional ANSI terminal emulator with a save/restore interface that expresses its state succinctly (relative to an arbitrary prior state), you can do the same tricks of syncing the server-to-client state at whatever interval you want and recovering efficiently from dropped updates.

For lambda, it's fun to imagine that every compute-intensive job you might run (video filters or search, machine learning, data visualization, ray tracing) could show the user a button that says, "Do it locally [1 hour]" and a button next to it that says, "Do it in 10,000 cores on lambda, one second each, and you'll pay 25 cents and it will take one second."

<comment> TD-Linux Many encoders have a "two pass" mode that saves statistics from a first, fast pass to guide the second pass. Usually these are very simple statistics, used to choose a quantizer and frame type in the second pass. Your method feels like a rather extreme version of this, that can guide the whole RDO search with a very large state. And, of course, rarely are first passes done in parallel. It's super exciting to see people looking at this - I think there's a lot of untapped potential. The WebRTC case looks interesting too, though there you need extremely low latencies (1 frame) so I'm curious to find out how you tackle that.

One downside with your approach is that you have to code many keyframes that are eventually thrown away. Keyframes are often expensive to encode because their bitrate is so much higher. Have you considered "synthesizing" fake keyframes somehow, such as doing an especially fast and stupid encode for them? Also, artifacts from keyframes tend to be a lot different from inter predicted frames, so would encoding a keyframe, plus a second frame, then throwing away both yield a quality improvement at the final pass (at the cost of more wasted computation)?

<comment><USER2> Thanks for your kind words! For the WebRTC/real-time case, here is our current draft ([https://cs.stanford.edu/~keithw/salsify- paper.pdf](https://cs.stanford.edu/~keithw/salsify-paper.pdf)). Would be eager for any comments or thoughts or ideas for experiments to run, as we have a few weeks to revise it before the final version is due.

Re: having to code many keyframes that are thrown away, in practice we're using vpxenc for the initial pass, and it's just a heck of a lot faster than our own C++ codec (whose benefit is that it can encode and decode a frame relative to a caller-supplied state). We then use our own encoder to re-encode the first frame of each chunk as an interframe (in terms of the exiting state of the previous chunk), and that one encode ends up being slower than encoding six frames with vpxenc (see figure 4). So in a real system, you're absolutely right that this is another optimization opportunity, but I think for our purposes there's a lot of other low-hanging fruit we'd want to tackle first.

</comment></comment></comment><comment><USER3> $5.40 to encode a 15-minute video is... a lot. Amazon Elastic Transcoder runs at $0.03/minute, so it would be $0.45 to encode the same 15-minute video.

The value of doing video processing with general-purpose compute is either A) you can adopt new codecs quickly (to save operational costs), or B) you can do processing in batch with older/slower/off-peak compute resources (to save capital costs).

If a company was going to spend $XM on engineers to build a low-latency video processing solution at scale, you'd go for the "remove-a-layer-of-abstraction" of thousands of tiny threads: purpose-built real-time encoding silicon.

<comment><USER4> This is about processing video in real time in the cloud. The intro mentions that in the nonparallel case, 1 hr of 4k video can take 30hrs to process. Doing it in <1hr has real benefits.

<comment><USER5> AFAIK transcode-as-a-service vendors already offer real-time/streaming transcoding, so it's over as soon as you finish uploading the file.

<comment><USER2> Not in our experience -- 4K and VR encoding is pretty hard. We uploaded the same video to YouTube (4K Sintel) and counting from the end of the upload, it took 417 minutes before it was available to play in 4K VP9, and 36.5 minutes before it was available to play in 4K H.264. (ExCamera took 0.5-2.5 minutes by the same metric, i.e. 6-30x faster than real-time.) You can certainly do "ready-as-soon-as-uploaded" encoding if the resolution is small, upload bandwidth is bad, and the codec is simple, but as networks get faster, resolutions bigger, and video encoders more complex, it gets more difficult.

Even a real-time hardware encoder isn't enough for the target use-case here, where the user already has the video in the cloud and wants to apply a transformation (like an edit or filter) and then re-encode the results. ExCamera is trying to make that as interactive/quick as possible. "Real-time" on an hourlong video would still mean waiting like an hour for each operation.

<comment><USER6> But you weren't paying for this as a service were you? You don't know that YouTube didn't put you in some queue and you don't know how long YouTube actually took to encode it, or how quickly they would be able to encode it if they wanted to? Or am I misunderstanding you?

<comment><USER2> We haven't tried a paid service, although my sense is that YouTube does represent the best-of-breed in commercially available parallel transcoding (free as it is to users). I'd encourage you to try it with a paid service and see where the time-to-finished 4K output shakes out. I'd be surprised if it's better but eager to learn either way. ExCamera is doing it at 6x-30x real time (which is important if you want to make a cloud video editor); I don't think you can buy that commercially today.

Ultimately, YouTube uses chunks of 128 frames each because it's too expensive to put key frames much more frequently than that, and other transcoding services will face a similar limitation. ExCamera's purely functional encoder lets it use chunks smaller than the interval between key frames, so the effective degree of parallelism can be a lot higher.

(Obviously there's a lot of overhead from using a software codec -- we think hardware codecs should support a "save/restore state" interface too. I think we do have to prototype first in software to demonstrate the benefits before hardware people will take us seriously.)

</comment></comment></comment></comment></comment></comment><comment><USER7> Nice approach. I assume this can be adapted to run in a single process on a multi-core machine to increase parallelism.

But...

Why no comparison to the currently established approach of encoding chunks of 5sec/128frames, all in parallel?

The paper mentions this approach, but only compares to single- and multi- threaded vpx. As a reader I'm left wondering if it is because this approach leads to a similar overall encoding time.

The paper also fails to mention that keyframes are also there in regular intervals to help seeking, especially for streaming video.

Also the way of measuring youtube "encoding time" is ridiculous. This probably measures the length of the encode job queue and currently available unused compute in gc2.

<comment><USER2> Thanks! Unfortunately we couldn't run 128-frame independent chunks (which we would call "ExCamera[128,1]") on an AWS Lambda instance because it doesn't have enough storage or RAM to store 128 frames of raw 4K video. We did benchmark ExCamera[24,x] (including the independent case, ExCamera[24,1]), which is the biggest we could fit on a lambda, and include those results in the paper. All of our tested configurations end up with a keyframe at least every 192 frames (the max configuration was ExCamera[12,16]), so we're sort of similar to the status quo in terms of keyframe (i.e. type-1/2 SAP) frequency.

<comment><USER7> Thanks for the insight.

The vpx runs were done on normal VMs. But I see that approaching a smaller runtimes, VMs are not economical.

The theoretical bound of splitting the source material into 5s chunks and encoding them all in parallel is still a useful comparison to make. Especially considering an endless supply of jobs keeping VMs busy.

</comment></comment></comment><comment> Const-me I wonder if it's possible to port to GPGPU? For some workloads, thousands of tiny threads work very fast on these. If yes, it might be much more power efficient and therefore much cheaper to run.

</comment><comment> TD-Linux This is really cool. They use VP8, which means that any web browser (except Safari) can decode the video, even over low-latency WebRTC. It also shows a lot of the limitations of VP8/VP9 encode slowness is due to libvpx, not the underlying format. It'll be interesting to see what they can achieve with VP9 or AV1.

</comment><comment><USER8> I think this paper is important in ways not related to applicability or cost efficiencies, but more as a what if we treated a warehouse scale computer a million core device? What if we programmed this device like simd or a gpu?

</comment><comment><USER9> Great paper! One thing that isn't mentioned is how do you deal with audio?

<comment> TD-Linux Audio encodes far faster than video, so there isn't really a need to parallelize it. You'd probably combine this with Opus in WebRTC.

<comment><USER9> Audio is fast however if you are attempting to encode hours of content ~min it's gonna be a bottleneck

<comment> TD-Linux If you end up being limited, there is a very similar trick you can do with audio, but even simpler. Unlike video, most audio formats don't have keyframes, but rather will converge to a correct decode a few packets after you start fresh or seek. So the solution to encode in parallel is to split the file into a bunch of chunks that overlap by a few packets (in the case of Opus, 80ms worth, or 4 packets, should be enough). You then encode all of these chunks, and then merge them together, throwing away the extra packets in the overlap. Unlike video, no final encode pass is needed.

</comment></comment></comment></comment><comment><USER10> I wonder how this fares now considering all that context switching that just got more costly from Meltdown mitigation.

<comment><USER4> Dubious. You make me more threads to make the same throughput, but that's the same as anyone else. Constant increases in overhead aren't hard to model or accommodate in many parallel throughput problems.</comment></comment>


Ask forgiveness, not permission - <AUTHOR> http://venturehacks.com/articles/ask-forgiveness-not-permission?utm_source=feedburner&utm_medium=feed&utm_campaign=Feed%3A+venturehacks+%28Venture+Hacks%29 <comment><USER2> Sheesh, guys. If you take every single piece of advice you read on a blog as a black-and-white gospel that you should never, ever disobey, then I fear for your ability to create.

Ask for forgiveness is to prevent you getting stuck into analysis-paralysis (as much as it grosses me out using business-speak like that, the term fits).

The top comment here as of now, about exposing a firm to $5b worth of risk?

C'mon... Do really think _anybody_ is _actually_ advocating that?

<comment><USER3> A lot of the time, using absolutes is pointless and irritating.

</comment></comment><comment><USER4> This is one of those things which sounds great, implies 'get it done' kind of people, and can have profoundly bad outcomes. Engineer testing auto-image- upload from a friends phone. Bad idea. Creating a release system where everything you copy into a magic directory, goes to the web, bad idea. Both sought forgiveness, both got fired. California being a 'right to work' state and all.

Asking permission might not be the right answer but checking your ideas out definitely beats just going for it.

<comment><USER5> There is not enough context to judge each action (ie such moves could be bad in specific markets with lot of red tape)

However, by default I'd say there are great ideas, and if they got the boot for asking forgiveness, maybe it's a good thing for your friends - as it it's better to be working where one's value is recognized, than at a company digging its own grave in the midst of a lack of feature, lack of ambition and lack of insight.

</comment></comment><comment><USER6> Every single company under the sun says that they follow this practice. But the reality is that very few companies actually follow through. Even startups who swear up and down they give their engineers complete control over what they build.

<comment><USER7> >But the reality is that very few companies actually follow through. Because asking your customers to pay you for a product/service when you also expect them to be your testers is going to lose you a lot of customers long-term in most markets. If something doesn't work consistently, or they are constantly finding new issues they will simply find another company to work with. Not to mention, by letting people push whatever they want to production and waiting for a customer to report issues you are counting on people to actually submit bug reports & not just ignore/abandon the issue.

</comment></comment><comment><USER8> Developers _shouldn't_ have this kind of control unless your product is _for developers._ I say this _as_ a developer.

The article mentions it only works if you hire insanely smart people, but that's also not a sufficient condition. You can have really amazing programmers, but if they don't have a strong understanding of your demographic and your goals, they're not "smart" in the way that's most relevant to this particular sort of decision.

I also dislike the phrase to begin with. Asking forgiveness rather than permission is often more effective, but that doesn't mean it isn't also irresponsible and a breach of trust. In an environment that _doesn't_ claim this sort of non-regulation (i.e. most of them), if your integrity sells for the value of one commit and the risk of breaking or misdirecting your employer's project, you have more fundamental problems than potential bugs in production.

<comment><USER9> _If you have the freedom to make decisions, you also have the responsibility of being correct._

I'm not sure of the full intentions of the article nor your comment, but I think you may have missed a major point here. The article does not seem to argue that insanely smart people know all the right answers; instead it argues that insanely smart people have a pretty good idea when something is a bad idea, and will be the type of person who works out the right answer prior to pushing it live. That said, the other cases is covered as well...

 _Actually, mistakes are fine. They're something you trade off for other variables like speed of iteration._

<comment><USER8> One problem I have with your second quote is that "speed of iteration," like intelligence of employees, is not in and of itself enough to make the result good for the product. The latter requires that the employees understand the target audience and their needs. The former requires that your iterations are moving in a direction that satisfies your target demographic, and if it doesn't, this isn't just not beneficial: Your speed of iteration becomes directly detrimental to the product.

</comment></comment></comment><comment><USER10> _This policy only works if you hire insanely smart and capable people, and let go of the ones who are not._

Yet another pretentious startup blogger. I am insanely smart and capable, yet for anything beyond a cute web app, I don't think you should use users as a debugging tool

</comment><comment><USER11> Right, the problem is most people who claim to do so NEVER ask for forgiveness...

OTOH I don't accept he sentence but that is a different question.

<comment><USER12> Indeed. Most of the time, it seems to me like it appeals to the people who read it as "Ask grudging resignation, not permission".

</comment></comment><comment><USER13> The article mentions it (briefly) but it's probably worth noting: Valve's a living example of success using this model. These guys went from zero to I-own-the-software-industry (both in development and distribution) in a breeze!

Employees get to pick their own projects: their work model is based around self-gathered work groups. Got an awesome idea? Go make a group, gather fellow employees and DO IT! The company is so centered around this fact that employee's desks have wheels so they're easier to move around.

Okay, this is just anecdotal evidence and they might just be the luckiest company ever, but they managed to pull the trick and it would be dumb to overlook their success.

Here's a cool handbook for new Valve employees if you want to take a deeper look at how Valve works:

[http://www.valvesoftware.com/company/Valve_Handbook_LowRes.p...](http://www.valvesoftware.com/company/Valve_Handbook_LowRes.pdf)

</comment><comment><USER14> Without controls in place, let's see how much "forgiveness" you receive from the SEC and Sarbox enforcers when things go awry and shareholders lose out.

</comment><comment><USER15> Venture Hacks blog is back? Oh... I hope so!

</comment><comment><AUTHOR> Would love HN wisdom! We're using Twitter to build an NLP app, and everyday we're wondering if they will just pull the plug on us. We have a friend at Twitter in Ops: ask permission or forgiveness?

<comment><USER16> The underlying question of your comment is whether this is a stable business model. You already indicate that Twitter has the _ability_ to disrupt/shutdown your service, which puts you in a difficult situation to build something of value. If you are ever wildly successful and dependent on one platform, then that platform will extract value from you or replace you. Consider this a rule, not a guideline, because it always makes business sense for the platform to do that. Recent examples include Zynga paying the Facebook "tax," Craigslist effectively shutting down PadMapper, and the slew of Twitter 3rd party apps that have been crippled by limited/revoked access.

</comment></comment><comment><USER17> Re: Don't fuck around.

</comment><comment><USER18> _Ask forgiveness, not permission_

Dear Board of Directors, During the past few weeks I have been engaging in unauthorized trading. I bet that the Euro would weaken drastically, hoping to make the company at least $4.6 Billion. Instead, we're 5 billion down, as the situation changed.

I was going to ask for permission to use all that cash to trade, but I read on a blog that I should ask for forgiveness, after the fact. So, forgive me.

<comment><USER19> The policy is not about _always getting forgiveness_ , it's about asking for it when you can realistically expect it to be granted. A company employing that policy should also be very clear about what acts you'll never get forgiveness for, and that would of course include breaking the law like in the parent's (obviously exaggerated) example.

<comment><USER18> BS, they are essentially asking people to take the risk, if it works _they_ benefit a great, if it backfires, they'll issue a press release disavowing you.

Remember, a hacker is not Google or Apple to get away with book scanning ("ask us nicely to remove scans of _your_ books") or location tracking.</comment></comment></comment>


Look Ma, No Glue: Electrostatic Adhesion as If by Magic - <AUTHOR> https://hackaday.com/2019/03/09/look-ma-no-glue-electrostatic-adhesion-as-if-by-magic/ <comment><USER2> I'm curious about what one of the comments asks. Anyone know the answer?

>How dangerous is this? If the insulation fails and you touch the plate, will you be killed directly or just receive a harmless shock like from static electricity?

<comment><USER3> I saw elsewhere that the power supply is from a bug zapper, so I'm guessing about like touching a bug zapper. A little more googling has told me that the answer is "it's bad, but not lethal or burning if you move your hand quickly"

<comment><USER4> Semi-relevant!

[http://amasci.com/weird/unusual/e-wall.html](http://amasci.com/weird/unusual/e-wall.html)

</comment></comment></comment><comment><USER5> The electrostatic chuck is a common way to hold silicon wafers in chip fab machines.

[https://www.sandia.gov/LabNews/LN03-27-98/chuck_story.html](https://www.sandia.gov/LabNews/LN03-27-98/chuck_story.html)

</comment><comment><USER6> video [https://twitter.com/JanHenrikH/status/1094664006059593729](https://twitter.com/JanHenrikH/status/1094664006059593729)

<comment><USER7> This demo is missing a key point: what happens if you open the circuit while the paper is adhered? Does it just fall down?

<comment> gene-h It will probably stick for a decent amount of time. An electroadhesive pad is a capacitor and leakage will probably be quite low.

</comment></comment></comment><comment><USER8> There are professional whiteboard style products that make use of electrostatic adhesion. Usually designed such that you stick a piece of paper behind a plastic overlay that you draw on. They are pretty nifty if you have a need for temporarily marking up printouts.

<comment><USER9> EEVBlog did a teardown of one: [https://www.youtube.com/watch?v=MsNbK7kCcKA](https://www.youtube.com/watch?v=MsNbK7kCcKA)

</comment></comment><comment><USER10> The Ping Pong Ball Accelerator referenced at the bottom of the article is a really neat project. Wonder if elem school would let the high voltage source be used during science fair demo day...

</comment><comment><USER11> Apparently, the source of high voltage came from an electronic bug zapper. Pretty neat.

<comment> gene-h Cheap ionizer power supplies from ebay and what not[0] can be used to do the same thing for $3-$10. It's best to find one that has an exposed ground wire, which can be difficult as most don't have documentation, but otherwise one can use one of the input supply terminals. I've found that a simply electroadhesive pad a bit bigger than a standard piece of paper with two electrodes made from a space blanket glued on to some polyethylene sheeting can support the weight of a power supply and a 9 volt battery to power it.

[0][http://amasci.com/emotor/list_ionizers.html](http://amasci.com/emotor/list_ionizers.html)

</comment></comment><comment><USER12> Version 0.0000001 of Starship Enterprise's gravity system... <g>

</comment><comment><USER13> I'm no chemist, but does glue work by electrostatic?

<comment><USER14> Not in this sense I don't think, this is a case where turning off the power to the device causes the 'sticking' to stop. Glues are chemically (and/or mechanically) bound to surfaces, and you can't 'turn them off' (though you may be able to chemically alter the glue bond, e.g. with a solvent).

<comment><USER13> I meant inter-molecular electrostatic forces that make glue work.</comment></comment></comment>


Ask HN: What are the unofficial (i.e., official) rules of HN? - <AUTHOR>

I've been on HN for roughly a year, so I've got a general idea of the "rules to live by" on HN; however, every so often I feel like I missed a huge rule. (I was one of the ones to post a TSA article. :] )
So, HN, what are the rules I must know, every new HN'er must know, and every old HN'er that doesn't pay attention must know to keep HN a place that is profitable and worth our time?
Here is a list I've come up with:

  1. Don't +1 comment; up-vote it. (Or other variations of "I agree.")
  2. Step away from the politics machine, or somebody could get hurt.
  3. Only comment if you have something of definite worth to add; avoid flamewars at all costs.
  4. Don't be stupid. HN doesn't like stupid. We downmod stupid.
  5. Do keep your comments as short as possible, though, the random 300 word comment that actually adds value with every word is perfectly legit.
  6. Do post interesting stuff, just don't flood the /newest page with them.
 What would you add/edit/delete from that list? <comment><USER2> Google the link you're thinking of submitting. Chances are, it's been submitted and discussed as least a half dozen times before you.

Also, you made no mention of <http://ycombinator.com/newsguidelines.html>

</comment><comment><USER3> <http://ycombinator.com/newsguidelines.html>

</comment><comment><USER4> Don't say anything on here that you wouldn't say in person. Civility is paramount.

</comment><comment><USER5> Consider downvotes as comments on the quality of your writing and edit your post to more clearly state your position.</comment>


Trakdot: Track your Luggage - <AUTHOR> http://www.trakdot.com/en

<comment><USER2> I've been a user since it was a Kickstarter project. It works quite well, actually. The device comes with a globally activated SIM card inside that sends an email/sms to you every time in you in a new location. There's also an Android/iPhone app that has a a bluetooth locator that you can pair with the device so you can track the proximity of your luggage inside the airport. The location is also shown on the app but it's just a webview of the website.

</comment><comment><USER3> The FAQ says: "...device emits a signal that is used to determine its location, but does not use GPS."

Anybody knowledgeable can throw light on what kind of signal it could be?

<comment><USER3> Ah never mind. It says later on "The device will not operate in countries where GSM does not function."

</comment></comment><comment><USER4> wow $50 is a solid price point and puts it in impulse purchase zone for a lot of people. I figured it'd be $100 or more. Not for me as I don't travel enough with checked bags, but still, props to the founders.</comment>


Ask HN: Building a side project that makes money. Where to start? - <AUTHOR> I work for a company in customer service department. In my spare time I have learnt a bit about HTML, CSS, Python. I can code in python with some help but my skills are not much good.
I want to build a side project. Especially, one that makes money. Even if it brings in $100, I feel that it will provide more satisfaction than my current job.
I don't have any frontend dev skills. Where should I start?
Should I outsource the website development part? I am 31.
Can you provide some ideas on where to start? What are some simple things I can build by myself? Any ideas? <comment><USER2> 1\. As others have covered, you need a market. Find something that interests you, or you won't stick with it. For me, I manage an events guide for local bands and pubs to compile their free-to-the-public gig listings. It's taken various forms over the last 15 years or so, from Mambo to Joomla to Wordpress to a Facebook page, and I'm finally getting around to rewriting it as a standalone product using Django, and it's only as part of this iteration that I'm monetising it is starting to look like a potential option.

2\. You may also need an audience. You need people to use your product, and that might mean you need some kind of free tier to lure them in. In my case, an events guide which doesn't list all the events people want to see is useless to my audience, and if the audience don't use it it's useless to my potential clients. That means I have to provide the basic service for free - any monetised features are worthless without it.

3\. For any skills you're lacking, learn them. If you outsource your front end work, you've got to pay somebody every time you want to change it and you can't reuse it anywhere. Once you know what you want to do, learn how to do it by actually doing it.

You're absolutely right that any money earned from your own pet project will be far more satisfying than your day job, but that's because it's that much harder to earn. If you start down the road with the specific intention of earning, that road will be paved with frustration. Get an idea, develop it into a prototype, get some feedback, and then you can start looking at outsourcing some of the more polished parts that are still outside your capabilities.

</comment><comment><USER3> I made colormind.io - it makes about 200 a month. Here are some concrete steps you can follow:

1\. market research: think about a field or niche you're interested in. Type some keywords into Google keyword planner to gauge search traffic, pick one with at least 10k hits per month. For me this was "color palette generator"

2\. build: type the keyword into google and check out the competition. What kind of content are they serving? Could you do better? You want to make something genuinely good, novel or interesting so people will link to you. For me this was the "AI" angle. Key point: use the exact keywords to refer to your site, to prime the discussion when others talk about it.

3: promote: post on relevant subreddits/producthunt/hn. If you're lucky a major general-interest blog or news source picks it up, otherwise you'll have to put in some legwork and do some link building.

4: monetize: it usually takes 6 months or so for organic traffic to trickle in, though it could be sooner. Once you have traffic, you can start to monetize. This is the easy part - google ads/carbon are good options. Affiliate links are less consistent - it depends on your niche and what the affiliate is.

if you're not good with design just use a nice looking free template.

</comment><comment><USER4> You'll find that's incredibly hard to make people use your side project even for free, let alone for money. Start small and consider it a win to have > 1 recurrent users who find your side project useful. Then you can think about monetizing it. But really, capturing user attention on today's Internet is a formidable challenge. Having a few recurrent users is an incredible achievement even if you don't make any money.

</comment><comment><USER5> I would recommend speaking to people about their problems. You will discover something eventually. Once you've identified a problem, then it's time to design the solution. You might not necessarily need front end dev skills - there are all kinds of apps that let non-coders piece things together so that it's a functioning solution.

Check out [https://bubble.is/](https://bubble.is/). Validate your solution and then when you need additional horsepower, it might be worth hiring developers. I've a friend who builds software with offshore resources. They do work for big names such as Spotify but also startups as well so happy to connect him with you.

For inspiration on various ideas, check out [http://indiehackers.com](http://indiehackers.com) \- a site primarily all about entrepreneurs bootstrapping side projects to profitability.

</comment><comment><USER6> My suggestion is to make something you would find useful but that don't already abundantly exist (don't make a todo app or a calendar). This way you won't feel frustrated if nobody uses it but you. Keep it simple, keep only the core and polish it.

For ideas, I try to look into where I spend time during a typical day/week/month and see if I can find a way to simplify it or automate part of it. Then see if that's an issue other people have too by discussing with your friends, colleagues or making online searches on the topic.

Don't get hung up on your idea if it's not useful to you. Try another one and write down your ideas when they come to you so you can revisit them later. The best ideas are usually the ones that keep coming back.

</comment><comment><USER7> Solve your own problem, its likely that there are a lot of other people who struggle with it.

Once you've solved it, focus on distribution - reaching the aforementioned people. Go where they are and communicate with them: HN, Reddit, Twitter, specialist forums, email.

Be ready to pivot quickly as you'll probably find that the product you made, while useful, isn't as useful as it could be.

</comment><comment><USER8> Find one person. Do something for them that they will pay money for. Find more people like person one. Make sure you aren't losing money.

Let the budget/timeline influence these two questions:

I don't have any frontend dev skills. Where should I start?

Should I outsource the website development part?

Not sure how this is relevant: I am 31.

</comment><comment><USER9> Forget about the money part for a while; look at problems to solve and things you can build as learning exercises to try and get your skills up to scratch. Look at a todo app - boring I know but you'll learn a lot.

I've built a few things I find useful and expanding on them to try and monetise them.

</comment><comment><USER10> I would suggest creating a git repository on GitLab or GitHub to get started. Then learn about nodeJs, npm and create a build project, storing it on your remote git repo (GitLab private repo). Once you have started developing these skills try submitting to an open source project, getting feedback. When you feel you know enough about the process, look for small gig work and build up your resume from there. Lots of perils, but worth it. Good prep for the business skills you need.</comment>